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(57) ABSTRACT 

A method and system for providing a telephone caller 
information assistance such as driving directions from a 
starting location to a destination location. To obtain infor- 
mation assistance, the caller places a telephone call to access 
the system. If the geographical location of the caller can be 
determined by an automatic location identification system it 
is displayed on an operator console where the caller's 
request is transferred to be handled by a live operator. The 
operator receives the caller's destination request and queries 
the system for the street route driving instructions to the 
requested destination. After obtaining the street route driv- 
ing instructions, the call can be transferred to an audio box 
having an interactive user interface capable of replaying the 
desired information to the caller. In an illustrative 
embodiment, the interactive user interface is capable pro- 
viding functions to stop, start, pause, and replay the infor- 
mation to the caller. 
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That is not a 
valid keypress. 



no 




71^ 



^ Playback paused. 
To resume playt)ack, say "resume" or press 5. 



130^ 



♦Note: Play this message every 30 
seconds. Play this message if bad 
speech commands are detected 
When bad speech commands are 
detected, do not play this message 
more than once every 10 seconds. 



13^ 



Hold for up to 5 minutes, ^ 




1,2,3A6,7,8,9,0,*,#-.. 



Your call is being transferred to 
the next available operator. 



3- 



6 



5, resume 




08/18/2003, EAST Version: 1.04.0000 



U.S. Patent Mar. 25, 2003 sheet 8 of 18 US 6,539,080 Bl 




08/18/2003, EAST Version: 1.04.0000 



U.S. Patent Mar. 25, 2003 sheet 9 of 18 US 6,539,080 Bl 




08/18/2003, EAST Version: 1.04.0000 



U.S. Patent Mar. 25, 2003 sheet 10 of 18 US 6,539,080 Bl 




08/18/2003, EAST Version: 1.04.0000 



U.S. Patent Mar. 25, 2003 sheet 11 of 18 US 6,539,080 Bl 



Command not recognized. 




Command not recognized 



Command not recognized 



Your call is being transferred to 
the next available operator. 



Transfer call to operator 



170 



3- 



7 
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Driving Instaictions 
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Step-by-step 
Driving Instructions 
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METHOD AND SYSTEM FOR PROVIDmG Needed is an improved system to infonnalion and travel 

QUICK DIRECTIONS directions, which is readily accessible to travelers. 

BACKGROUND OF TOE INVENTION ^^'^ DESCRIPTION OF TOE DRAWINGS 

A. Field of the Invention ^ foregoing and other objects, features and advantages 

The present invention relates to a method and device for P^^^^^ invention will be more readily appreciated 

providing information assistance to telephone callers. More ^P°° reference to the foUowing disclosure when considered 

particularly, the present invention is capable of providing ^ conjunction with the accompanying drawings, in which: 

information assistance based on an automatic identification FIG. 1 is an overall system diagram of a system; 

of the caller's location and providing an user interface to pjo. 2 shows an illustrative diagram of a call detail record 

communicate the information over the telephone to the used in the system of FIG 1- 

"a Description of the Related Art . ^IG- 3 ^^<^ an diagram of the generation of route 

„ , * 1 J „• . A 1 *■ u instructions used in the system of FIG. 1; 

Today, travel directions to a destination location can be ^ 

obtained in different ways. Internet web sites now provide F^^S. 4 through 12 are command menu charts of an 

map routing software that generate and display driving illustrative embodiment of the interactive user interface used 

instructions from one starting location to another destination the system of FIG. 1; and 

location. The traveler enters the desired starting (from) FIG. 13 is a flow chart of an illustrative method used in 

location and the desired destination (to) location into the the system of FIG. 1. 

software program and has the option to select the fastest, 20 

easiest, or shortest distance driving directions to the desti- DETAILED DESCRIPTION OF THE 

nation location. The map routing software determines the PREFERRED EMBODIMENT 

route to the destination and displays it on the user*s display ™ * i. j- . n . i ^ n . i 

I * . , J • • J- *• * J The present embodiment allows telephone callers to place 

terminal with step-by-step driving directions, estimated n. . 

travel times, and mileage Lm the starting location to the 25 ' ^f^^T ^""i ' ^.^^'"''^ -^formation 

ending location. This type of map routing software, and assistance m locatmg selected destination loca^^^ 

however, requires a traveler to plan his trip in advance and step-by-step driving du-ections to reach the selected 

have Internet access. It is of little assistance in the situation destination locations over the telephone. An illustrative 

when a traveler becomes lost during his trip, needs assis- embodiment provides the ability to automatically identify 

tance with directions, and does not have ready access to the 30 ^he calling party's calling location without requiring the 

Internet. caller to manually identify his location. The calling location 

Recently, Global Positioning Satellite ("GPS") navigation ^j^^" "^^^ °^^P ^ destination location 

systems allow a traveler to track his location in real-time as determine the proper driving instructions for the caller 

he travels on the surface of the earth. In-car GPS-guided ^^"^^ requested destmation location, 

navigation systems allows an automobile driver to track his 35 1° ^ descnbed embodiment of the invention, telephone 

current location in real-time and obtain voice and visual callers place a telephone call to access the system and select 

directions on how to get from his current location to a ^ desired destination location. The destination location may 

destination location. Destination locations can be selected selected from a white or yellow pages listing through an 

from an in-car terminal by street address, street automated interactive voice response system or a live human 

intersections, or city. These on-board navigation systems 40 operator. The destination location may be any locations such 

also include categorical search tools that allow users to find ^ person or individual, or a street address, hotel, stores, 

points of interest such as museums, parks, airports, stores, restaurant, business office, etc. listed in the white or yellow 

etc. To reach his destination, the driver is given specific pages directory. 

driving instructions from an in-car terminal on how far to After selecting the desired destination location, the caller 

proceed and when to turn and exit. These on-board naviga- 45 may receive driving route instructions to the destination 

tion systems, however, are complex and expensive, require location from the system. For example, the caller's location 

a considerable up front investment and a complex installa- and the desired destination location is sent to a database with 

tion procedure. application software which can map a route and provide 

Today, many wireless telephone service providers also driving instructions from the caller's location to the desired 

provide a telephone service that allows users to dial *'*nnn" 50 destination location or from another location to the desired 

or a 7/10 digit telephone number to reach a live operator who destination location. The database and application software 

can provide expressway travel times, weather conditions and maps the driving route and determines step-by-step driving 

driving directions. The callers specify their current and instructions to reach the destination. In a particular 

future locations by providing the nearest cross street inter- embodiment, the route to the destination location can be 

section or street address. Destinations can be selected by 55 mapped taking into account the route traffic, travel-times, 

white or yellow page listings, street address, or cross street road conditions, and route weather conditions. 

intersection. The live operator also has categorical search The caller may receive the driving or route instructions in ] 

tools that allow callers to find points of interest. This service, a variety of different ways. The route instructions can be 

however, requires the caller to specify their current location communicated directly over the telephone from an interac- 

and is therefore not useful to those who are lost and 60 tive voice response system, a live operator, a synthesized 

uncertain of their current location. voice, a voice mail message, and Internet electronic mail, an 

In addition, the retrieval of driving instructions over the alpha/numeric pager or telephone or a Personal Digital 

telephone requires a live operator to relay the driving Assistant ("PDA"). 

instructions to the caller. The caller must transcribe each In another aspect of the present embodiment, an interac- 

sequence of the driving directions while the operator waits 65 tive user interface to provide access to information over the 

on the telephone, thus reducing the productivity of the telephone is described. The user interface is particularly 

operator. adapted to provide a convenient interface so that a user can 
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obtain route or driving instructions to a destination location handle obtaining the driving instructions. Alternatively, the 

over a voice telephone call. The user interface provides a system may be setup such that all the 411 operators are 

number of commands or specified keys to allow a listener to trained and equipped to handle the calls requesting driving 

easily stop, start, review, and skip through the step-by-step instructions. 

route driving instructions. The route instructions may be 5 The switch 16 answers the incoming telephone call arriv- 

coded to demarcate points that are convenient points to stop ing on the access telephone number and forwards the call to 

and start the instruction messages such that the user can the operator console 18 where a live operator can handle the 

listen and record the instructions or listen and pause the call. The switch 16 can be any suitable switching equipment 

instructions while they are actually driving the route. The such as an ACD, DMS series switch from Nortel, #5 ESS 

user interface provides a convenient mechanism for callers 10 ^^m Lucent Technologies, Galaxy 500 or other Private 

to obtain the route instructions, as well as other information Branch Exchange ("PBX") switching equipment. The 

over the telephone switch may also be embodied as an automatic call distributor 

FIG. 1 shows a' system level diagram of an illustrative f^^Ss^atT^^'^^ 

embodiment of the present invention. Shown in FIG. 1 are sequences a receive . 

a telephone subscriber or caUer 12 accessing the system as 15 ^^'^^^'o^^- " call detail record data structure is also 

a caller seeking assisUnce, an access or switching network created and sent to the op^ator consoU 18 dong with the 

, ' o i_ J 1 L XT . 1 forwarded telephone call. The call detail record may contain 

14 such as the Public Switched Telephone Network • 4 c «• *u • • * i 1 n 

/^.T^orr-vrm • -i . i i i i i u a vancty of informatioH regarding the amving tclcphoue call 

("PSTN ) provided by local telephone compames such as ^^^^^ ^ displayed on the operator console 18 to 

Amentech and long distance came^ such as /^ScT and p^^^^^^ ^^^^^^^^ information regarding the incoming 

WorldCom/MCL A switch device 16 such as a DMS-200 20 telephone call. In the illustrative embodiment, the call detail 

from Nortel or an Automatic Call Distributor ("ACD") such ^an include a number of pre-populated fields to show the 

as those from the Rockwell Corporation, which provides operator a number of details regarding the call such as the 

access to and routes calls through the system. Also shown is calUng telephone number, the called telephone number, the 

an operator console or workstation 18 where a human location of the caller, and the station type of the calling 

operator to handle calls may be located, and a database 20, 25 telephone number. Different station types such as coin or pay 

which may include a plurality of databases. In an illustrative station, residence, or hotel telephone systems are well 

example, the database 20 may include a directory listing known to those skilled in the art. For example, in an 

database 22, a GEO TN database 23, and a routing database illustrative embodiment the calling location of the incoming 

24. Also shown is a gateway 26 interface that receives telephone calling is determined using a suitable calling 

driving route instructions generated from the routing data- 30 telephone location technology 17 as will be described in 

base 24. The gateway 26 interface translates the text route further detail below. The location of the caller determined by 

directions to interface the route directions to the audio box the ANI is sent to the call detail record to the operator 

28. For example, the route instructions may be generated in console 18. 

a text format that the gateway 26 interface translates to a The operator console 18 displays the call details to the 

data format appropriate for the audio box 28. The appropri- 35 operator while the call is being handled. Referring now to 

ate data format may be a specialized text format that can be FIG. 2, shown is an exemplary display of the operator 

forwarded on to the audio box 28. The audio box 28 console 18 while handling a call requesting directions to a 

translates the route instructions into audio speech that can be destination location. It shotild be understood that FIG. 2 is 

heard by a caller using speech synthesis. The audio box 28 a simplified illustrative drawing and the operator console 

allows a caller to retrieve directions without the continued 40 display can be implemented as a Windows-type interface. At 

interactive assistance of a human operator. the bottom of the screen, a call detail window 32 can display 

Described below in connection with FIG. 1 is an overview call information from the call detail record, such as the ANI 

of an illustrative embodiment of the system in operation. A or location where the caller and the desired ANI where the 

telephone caller or user 12 desiring assistance with a variety caller wishes the directions to start. Often the caller's ANI 

of services such as travel directions, maps, weather, trafEc 45 and the desired ANI will be the same. The caller may also, 

travel times, directory assistance white page listings, and however, request that directions be given from a different 

road conditions originates a telephone call to a dial-in location, resulting in a different desired starting location, 

telephone access number to access the system 13. For The address fields and the cross street information can be 

example, a landUne or a mobile cellular telephone subscriber determined from the ANI and the database information. The 

12 dials the telephone number of the dial-in access telephone 50 station type of the caller may also be displayed, but is not 

number The telephone call is routed through the switching explicitly shown in this example. The operator views the call 

network 14 to the dial-in access telephone line terminated to details and queries the subscriber 12 for his or her request, 

the switch 16. The switching network 14 may include the Preferably, the caller's location is automatically identified 

PSTN as previously described. The dial-in telephone access and displayed on the operator console 18 as described above, 

number is preferably a double digit, 7-digit, 10-digit or toll 55 Of course, the operator can request the caller for the starting 

free telephone number such as an "800" or "888" telephone location to confirm the automatically identified location. The 

number To provide greater call capacity, a high-capacity operator can accept the information in the call detail window 

telephone line, such as a Tl or Primary Rate Interface circuit 32 or make any changes or correction to the information 

may be used to implement the access telephone line. Wire- before making the request to the database to map the route, 

less telephone users may be configured to simply dial a 60 The caller's request is entered into the operator console 18 

"*NNN" to access the system. In a preferred embodiment, or if the caller wishes to search for a destination location, a 

the service may be accessed through a basic "411" directory search is launched against a listing database for candidate 

information service. For example, callers may dial 411 to listings according to the caller* s request. In the illustrative 

obtain directory information and also be given the option to embodiment of the invention, the listings that can be 

obtain street driving instructions to the requested directory 65 searched include desired destination locations such as street 

listing. If driving route directions are desired, the telephone addresses, hotels, stores, restaurants, business offices and the 

call may be routed to the appropriate operator console to like that may be listed in the white or yellow pages. 
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If the desired listing(s) is found, the operator quires the techniques can be used depending on the particular tele- 
caller 12 whether directions are need to the selected desti- phone (i.e., landline or mobile telephone) originating the 
nation location or for any other location the caller 12 may be call. Several exemplary telephone caller location identifica- 
interested in obtaining routing information. If the caller tion systems 17 are described below in further detail. The 
wishes to receive directions to the destination, the operator 5 location of the caller 12 can then be communicated to the 
uses the starting and destination locations to determined an operator by including the geographical location with a call 
appropriate driving route between the locations. ^^^^^^ ^^^^^ structure that is transferred to the operator 
' ^ J. 1 . . console 18. 

The starting and endmg locations are be applied to a „ i .i. i c . i i. n r 

j.L jji • r . For example, the location of a telephone caller 12 from 

database 23 contaimng Geo-oodcd location information to , ... i 45 1 , , j i- i_ -j j 
. , . u • 1 I.' J- * r *i_ J * J in traditional fixed location land-lmes can be identified using 

obtain the physical geographic coordinates of the desired 10 ^. , . u a<aktt»\ i- .1. • * 

1 TT. L • 1 u- J- . n *i. the automatic number identification ( ANl ) of the onei- 

locations. The physical geographic coordinates allow the . n- * 1 u i_ * • ^ • . j- . 

u- I 1: c f-** J J *• *■ 1 * natmg calling telephone number to index mto a duectory 

geographic locations of starting and destination locations to i-.* j*u kt a^j jt-ii. i_ 

u J * • J A J J J . u J L- listing database or Name, Address and Telephone number 

be detennined. A geo-coded database provides geographic . ^ a*xtatt-»»\ r *i. antt r.i. «• .1 

I * * 'Zee Database ("NATO ) 22. From the ANI of the calhng tele- 
vertical and honzontal location coordmates in the form ot . V .u ktatt^'^-i i_ j r .t. jj 
i^.^j ji .^j _j-.r J L phone number, the NATD 22 can be accessed for the address 
latitude and longitude coordinates for nodes such as street ^ • • !• . 1 u t_ ^m. • • • 1 

,j J • n . 1 J 1 of the onginating telephone number. The onginatine tele- 
addresses and significant landmarks. i_ J.J 

phone number address can then be used to determme the 

Referring to FIG. 3, the starting and destination locations geographic location of the telephone number by accessing 

are appUed to a database 20 to map a route and create Geo TN or Geo-code database 23. 

step-by-step route driving instructions between the starting exemplary system 14. automatic number identifi- 

location and the selected destination location. Preferably, a ^^^^^^ ^ ^^^^^ 1^^^^.^^ identification 

map routing software program 21 applies the geo- ^^^^ .j^^^.^ originating calling telephone 

coordinates ot the selected locations_ to the routing database ^^^^^ ^^^^^ 

is passed in the call detail record to the 

24 to determme appropriate routing instructions 25 from the ^ ^^^j^ ^ illustrative embodiment, the 

startmg geo-coordinate to the ending geo-coordmate as will ^ ^^^^ ^^^j^ ^^^^ jgj^ ^^^^^^^ Instructioa 

be described in more detail below In the lUustrative Computer ("RISCO with the AIX operating system such 

embodiment, the route mstructions or dnvmg directions 25 -^^^ assistance platform from the 

are step-by-step street driving instructions. The route Corporation. The operator console 18 can then access 

mstructions 25 can then be passed to the operator console 18 ^^^^^^^^ ^2 to return the street address associated 

for the operator to view and r^lay to the caller. Refeamg originating telephone number. Preferably, the NATD 

agam to FIG. 2, the route instructions can be displayed on ^^^^^^^ ^2 contains the street address of all the telephone 

the main portion or window 34 displayed on the operator ^^^^ j^^^^^^ contiguous United States. Such a NATO 

console 18. Preferably, the mstructions are demarcated m a ^2 can be provided in conjunction with the nationwide 

step-by-step manner as dlustrated in FIG. 2. directory assistance platforms operated by the local 

To coramumcate the route mformation 25, the operator exchange carriers or other telephone service providers. In 

may simply recite the route instructions to the caUer, or the preferred embodiment, the NATD database 22 also 

alternatively may release the route instructions 25 to the contains physical geographic location information for each 

text-to-speech audio system to automatically play the telephone line in the database as implemented in the Geo- 

instnictions to the caller. The text-to-speech audio system ^ode database 23. For example, each telephone line listing 

preferably includes an audio gateway 26 and audio transla- ^ay include physical location or geographic coding ("geo- 

tion box 28 suitable for translating text route instructions 25 coding") parameters such as physical x, y coordinates which 

to synthesized speech. In addition, the audio translation box ^an be utilized to identify the address and latitude and 

28 preferably includes an interactive user interface 29 that longitude coordinates of the originating telephone line. The 

allows a user to control the playback of the driving route physical location of a fixed land-line telephone call can be 

instructions. Using the interactive user interface 29. a user determined using ANI to identify the calling telephone 

can receive the route instructions 25 and control how the number and then using the calling telephone number to 

route instructions 25 are played back without the assistance i^dex into the NATD database to access the geo-code 

of a live operator at the operator console. Use of the audio location of the calling party as described above. Preferably, 

box 28 to provide instructions to the caller also improves the significant landmarks such as cross-street intersections, 

productivity of the human operator. The operator can pass interstate highway exit numbers, local communities are also 

the call on to the audio translation system to communicate added to the NATD database with x, y geo-coding. 

with the user and proceed to handle the next caller. telephone number of the caller 12 is matched 

Described below is a description of an illustrative by the NATD database 22, the NATD record with the 

embodiment of the present invention providing a more geo-code of the caller's location is included in the call detail 
detailed description of the particular components. 55 vvhich is passed on to the operator console 18. If the caller's 

Referring again to FIG. 1, to initiate access to the system, calling location cannot be automatically determined, a mes- 

thc caller 12 preferably originates a telephone call to a dialup sage is indicated in the call record sent on to the operator, 

access telephone number. Telephone calls are primarily The caller's location must then be manually identified 

originated from either traditional fixed location landline through interaction with the human operator that the caller 
telephones or mobile wireless telephones. Mobile wireless 50 location was not available. 

telephone may include cellular, personal communication in another illustrative embodiment with wireless mobile 

systems ("PCS"), satellite telephones, etc. telephone, the callers 12 may be mobile and their location 

After the telephone call is answered by the switch 16, the will most likely vary with each telephone call. In this 

geographical location of the telephone caller is determined illustrative embodiment, an automatic location identification 
using an appropriate caller location system 17 capable of 65 ("ALI") technology 17 is preferably used to determine the 

identifying the geographical location of the caller. To iden- location of wireless telephone callers. The most common 

tify the geographical location of the caller 12, several type of ALI technology 17 for mobile wireless telephones 
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Utilizes a network-based approach that applies mathematical operator conduct a search for a selected destination locations 

algorithms to the attributes of the wireless telephone radio in a variety of different ways. For example, the caller may 

frequency ("RF") signals received at a base station of the be seeking the restaurant or grocery store nearest his calling 

wireless network. Such location algorithms using the precise location. Using the location identified by the ALI, the 

timing in which the RF signal arrives different base stations. 5 operator can launch a database search for any such estab- 

the angle of arrival of the signal, and the amplitude of the lishments in the proximity of the caller. In an illustrative 

signal are well known to those skilled in the art. To analyze embodiment, the destination location is selected from a 

RF signals, additional electronic hardware to analyze and white pages listing database that includes complete address 

locate the origin of the signals is required at the base information, x, y coordinates and geo-codes for each listing, 

stations. The main advantage of a network-based location The address information preferably includes street, city, 

technology is that it can be utilized with existing networks state, zip code, municipality, community, and the county of 

using existing telephone wireless handsets. A separate net- the listed locations. In addition to simply identifying the 

work to locate the caller and new caller handsets are not listing, individual listings in a database 20, such as a white 

required. pages listing database, can preferably be selected or cat- 

Alteraatively, ALI of mobile wireless telephone calls may jj egorically searched in many different ways. For example, 

be implemented with a dedicated radio location network the white page listings database may include a search engine 

using a separate infrastructure and different frequencies than that can search the database by street, city, state, zip code, 

those used by cellular, PCS, paging, mobile satellite or SMR municipality, community, county or any combination of 

carriers, to determine the location of the caller. A dedicated fields. Appropriate search engines are available from a 

location network is employed to triangulate the position of 20 variety of sources including the IBM Corporation. The 

the caller though an analysis of RF signals received at database 20 may also allow the selection of listings by 

different points in the location network. Presently, the pri- category type of the listing, i.e., restaurants, gas stations, 

mary frequency band used for radio location is known as the stores, libraries, airports, etc. In addition, the database 20 

location and monitoring services ("LMS") band at 902-928 may also be searched for locations within a specific prox- 

MHz. An example of an ALI technology using a dedicated 25 imity or specified distance from the caller's location. In such 

radiolocation network is provided by Teletrac, Inc. The a proximity search, the geo-code location of the caller is 

Teltrac system is available primarily in major metropolitan identified and the geo-codes of the categories of entries are 

areas for fleet management solutions such as automatic searched for those entries within the desired location. In 

vehicle tracking ("AVL") and related assets tracking ser- addition, various combinations of the different criteria may 

vices. In addition, ttie most prevalent ALI technology today 3Q be utilized to search and select white page listings, 

is Global Positioning Satellite ("GPS") technology. GPS In a particular embodiment, callers may request informa- 

technology uses a network of 24 earth orbiting satellites to tion from the operator and select destination locations. The 

track a GPS receiver's location on the earth with an accuracy operators can access and search the database 20 in accor- 

between 60 and 300 feet. The GPS receiver's location is dance with callers 'requests to obtain and view the search 

determined by triangulating a timing and distance measure- 35 results. The operator can also interactively relay listings, 

ment between a plurality of the earth orbiting GPS satellites search results, and step-by-step driving instructions back to 

and a GPS receiver on the surface of the earth. GPS location callers 12. 

frequencies are provided by U.S. Government at well- Iq an illustrative embodiment of the invention, it is 

known frequencies. Location technology equipment is avail- contemplated an automated interactive voice response sys- 

able from a number of manufacturers such as Rockwell, tem provided by the audio box 28 allows callers 12 to access 

Trimble or Navtech. the information without requiring a live human operator. 

After the originating geographic location of the incoming Alternatively, a user may access the system using a com- 

call is determined, the call is sent to an operator console 18 puting device with a modem and a computer display. Pref- 

to allow callers 12 to specify their desired destination erably the computing device is a portable computer such as 

location. Switching device 16 switches the incoming tele- 45 a handheld or pahntop computer which can be used by a 

phone call to the operator workstation 18 to connect the call caller who is traveling and away from the office. The 

with a live operator. The operator workstation 18 will portable computer may have any suitable interface and 

display the ANI and calling location of the telephone caller display for showing text and preferably even graphics capa- 

12 to the operator console 12 if the ALI has determined the bility for displaying maps. The portable computer may have 

caller's location. If AU could not identify the caller 50 a Windows CE, Palm OS, Apple Newton or other operating 

location, the call detail will not show the caller's location on system suitable for a portable computer. Suitable computing 

the operator console 18 and the live operator must query the platforms include portable devices such as a Palm Pilot, 

caller to manually obtain the originating calling location. Apple Newton, portable Windows CE machines, or similar 

After the location of the caller is identified, the caller portable machines from Psion, Phillips, Hewlett-Packard, 

location is included in a call detail that is sent to the operator 55 and other manufacturers of portable computing devices, 

console 18 and displayed to the operator. Because a caller's These portable-computing devices can be used to access into 

location has been determined, the operator need not query the system on a dialup telephone line that can be provided 

callers for their location unless there was problem with the to allow access by a computer terminal. Once accessed to the 

ALI. The operator can then move on to immediately query system, the user can operate the system through a computer 

callers for their requests, thus speeding the handling of calls 60 interface, without requiring an operator 

and increasing the productivity of the operator. After determining the caller's location and identifying the 

The present embodiment can provide a variety of services caller's desired destination location, a route between the two 

for callers 12 such as those provided by directory assistance, locations can be mapped and step-by-step driving directions 

weather, time of day, yellow pages assistance, etc. For generated according to the mapped route. The map routing 

example, in an illustrative embodiment a caller 12 may wish 65 software typically includes a database 20 such as a route 

to obtain travel directions from his current location to a information database 24 storing map routing information, 

selected destination location. Callers 12 may request the The map routing software is applied to the database 20 
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preferably containing street map or routing information to 
derive the appropriate route between the locations. The map 
routing software takes a starting location and a destination 
location and maps a route between the two locations using 
the geographic location coordinates of the starting and 
destination locations. The mapped route can then be used to 
generate concise step-by-step driving or route instructions 
from the starting location to the destination location. The 
total distance of the mapped route and estimated driving 
time and estimated time of the caller's arrival can also be 
provided. Preferably, the mapping software converts cast, 
west, north, and south directions into left and right turn 
instructions. Map routing application software capable of 
generating driving directions are well known to those of skill 
in the art. For examples, such software can be found on the 
Internet at sites such as Mapquest, Expedia, Citysearch, and 
Mapblast. 

In a particular embodiment, the mapping application 
software can be provided in conjunction with a variety of 
real-time information such as weather, trafSc travel times, 
and road conditions. For example, local transportation 
authorities offer real time traflSc information on the local 
highways and interstate roads. These systems typically pro- 
vide travel times between selected locations as well as the 
speed of moving trafBc at road sensors embedded into the 
roadway monitoring the speed of moving trafiSc. Road 
construction information is also provided by the location of 
the construction (i.e., "Eastbound 1-88 at the Fox River 
Bridge")* the construction type (i.e., "Road Closure" or 
"Lane Closure") and the duration of the closure (i.e., "Feb. 
20, 1998 08:00 to Feb. 20, 1998 14:OffO- An example of this 
type of information is provided by the Gary-Chicago- 
Milwaukee ("GCM") Priority Corridor - Illinois Department 
of Transportation, Indiana Department of Transportation, 
and Wisconsin Department of Transportation in cooperation 
with the University of Illinois at Chicago Department of 
Electrical Engineering and Computer Science. The federal 
government has provided funding for these types of projects 
in the IVHS Act of 1991 as part of the Intermodal Surface 
Transportation Efficiency Act of 1991. Additional informa- 
tion can be found at the GCM web page on the Internet. The 
traffic/construction information can be used by the system to 
avoid routes that are closed or are experiencing undue delays 
due to construction. The system can be programmed to avoid 
mapping routes through construction routes by creating 
alternative routes, or recalculating routes not using the 
sections of road under construction. 

The map routing application software generates text route 
instructions in a format that is human or machine-readable. 
The text route insUiictions can then be transmitted to the 
operator console 18 to allow the operator to view the route 
instructions. The operator may then relay the route instruc- 
tions to the caller. 

The mapped route and step-by-step driving directions can 
be conveyed to callers in a variety of different ways. Of 
course, the step -by -step travel directions can be retrieved by 
a human operator and relayed to a caller by having the 
human operator read the directions over the telephone to the 
caller. Requiring a live operator read driving route 
instructions, however, may be cumbersome and require the 
operator spend a long amount of time with each caller. The 
operator must read the directions and wait for the caller to 
hear and record or transcribe each step of the directions 
before the operator can release the call and proceed on to the 
next caller. The amount of time spent with each caller is thus 
relatively accordingly long, reducing the number of calls the 
operator can handle, and requiring a large number of opera- 
tors to handle the expected volume of calls. 
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Callers accessing the system using a portable-computing 
device may download the route instructions to their com- 
puting device and review the instructions as necessary. The 
route instructions can be communicated over a telephone 
5 line using standard telephone modems, without requiring the 
interaction of an operator. 

In another embodiment, callers receive the routing 
instructions over the telephone without requiring a portable- 
computing device or the interaction of a human operator. In 
10 this particular embodiment, the operator console 18 sends 
the text routing instructions from the route database 24 to an 
audio gateway 26 that interfaces the text instructions to a 
data format suitable for the audio box 28. The operator 
console 28 also transfers the telephone caller 12 to the audio 
15 box 28, which can then provide an interactive user interface 
to communicate the route instructions over the telephone. 

For example, after retrieving the driving directions, the 
operator console 18 transfers the call to audio box 28. The 
audio box 28 provides the interactive user interface system 
to communicate route instructions to the caller 12. 
Preferably, the interactive user interface provides an inter- 
active voice response system with speech recognition to 
implement an automated user interface to provide informa- 
tion to callers. The audio box 28 may include an Interactive 
Voice Response ("IVR") system having speech recognition 
and speech synthesis capability. Suitable audio speech boxes 
are available from the Lucent Corporation which can be 
programmed with the assistance of the descriptions herein. 

The interactive user interface communicates the driving 
instructions to the callers and provides callers the abihty to: 

Pause and continue messages 

Back up to previous steps or messages 

Skip ahead to successive steps in the directions 

Spell street names 

Get instructions about using the system 
Repeat instruction from the beginning 
Return to a live operator 

40 The system allows the caller to speak commands or use 
the touch-tone keypad on the telephone handset to enter 
commands to control the playback of the directions. 
Preferably, the interface supports dual tone multi-frequency 
("DTMF") input and speech recognition with word spotting 

45 and barge-in capability. For example, after every menu and 
at each step of the instructions, a prompt tone is played to 
users to indicate the availabihty of the speech recognition 
commands. Preferably, the DTMF commands are always 
available to the user. The user will be allowed to perform 

50 actions with the DTMF commands while listening to 
instmctions for using the system. Announcements and error 
messages are inlerruptible by the DTMF commands. Error 
messages, however, should not be interruptible. Preferably, 
the DTMF interface also supports a type-ahead buffer to 

55 allow users to move through the menus and select com- 
mands and options without waiting for the system prompts 
through the menus. 

Travel directions may be given to callers using speech 
synthesis with a specialized vocabulary provided by the 

60 audio box. The route instructions or directions are given in 
a sentence-by-sentence manner and segmented or demar- 
cated by the system on a sentence-by-sentence or tura-by- 
turn basis. Preferably, the directions are demarcated at each 
point where the user is given an instruction to turn onto a 

65 new street in the route. At each demarcation point, users will 
be offered a command to repeat, pause, continue, backup, as 
well as other options. 
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In the illustrated embodiment of the invention, the 
instructions and commands for a caller retrieving directions 
to a destination location arc shown in FIGS, 4 through 12. 
Callers are ui^gcd to pull off the roadway before calling the 
system for assistance. However, it should be anticipated that 
callers may access the system while driving (against their 
better judgment), often while simultaneously attempting 
operate a motor vehicle and not have their hands free for 
manipulating a keypad. Thus, while receiving directions it is 
desirable that the caller can control the playback of direc- 
tions with vocabulary of voice recognized verbal commands 
using the speech recognition described herein. 

FIGS. 4 to 12 show flow charts of the possible menu 
selections at various points in the interactive user interface 
menu. 

Referring to FIG. 4, shown is a flow chart of an intro- 
ductory statement played to a user allowing the user to 
obtain further instructions. At step 102, the users may 
verbally request "instructions" or press the star key "*" 
followed by the "0" key to hear instructions about using the 
system. The interactive user interface includes speech rec- 
ognition to recognize the users'verbally spoken commands 
to initiate the appropriate menu selection. The speech rec- 
ognition is programmed to recognize a number of spoken 
key words that correspond to various menu selections. At 
step 104, the user may select the option of being transferred 
to a live operator by speaking a key word such as "operator" 
or pressing the "0** key. At step 106, pause of the playback 
of the system can be selected by saying "pause" or entering 
the "5" key. At this particular menu level, selecting the pause 
feature will paiise the system for about 2 seconds. Playback 
may also be resumed by entering any key selection. At step 
108, the information, which may include the route 
instructions, are played to the user. Step 109 represents the 
route instructions being played to the user 

Shown in FIG. 5 are several possible menu selections 
available to users while the route instructions are being 
played such as at step 109. As shown in FIG. 5, users can 
start the playback of the retrieved directions over from the 
beginning by stating "start over** or by pressing the "1" key 
on the keypad at step 110. After hearing each step in the 
directions, the user can say "continue" or press the "2" key 
to hear the next step of the route instructions at step 112. At 
step 114, the user can pause the playback of the directions 
by stating "pause" or by pressing the "5" key. Playback can 
be resume by saying "resume" or by pressing the "5" key 
again. The menu selections available from the pause features 
will be described in more detail below. 

Referring to FIG. 6, shown are the available menu com- 
mands from the pause feature selected at step 114 of FIG, 5. 
In this illustrative embodiment, a message indicating the 
instructions have been paused is played every 30 seconds. At 
step 130, the playback can be paused for up to 5 minutes. At 
step 132, the playback of the instructions is resumed by 
requesting "resume" or entering the "5" key. At step 134, the 
number of invalid verbal or key entries are counted and 
maintained. In the pause menu of FIG. 5, the "resume" or 
"5" key are the only valid entries. All other key entries (1, 
2, 3, 4, 6, 7, 8, 9, 0, *, #) or verbal commands are invalid. 
If speech that cannot be interpreted as valid commands are 
detected, the pause message is played no more than every 10 
seconds. Invalid entries are indicated to the caller. At step 
136, if the number of invalid entries during the pause feature 
exceeds 3 consecutive errors, the user may be having some 
difficulty with the system and is transfenred to an operator 
console for assistance. 

Referring again to FIG, 5, at step 116, requesting "spell- 
ing" or entering the "3" key causes the current step of the 
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route instructions to be repeated in a spelling mode by the 
user. The spelUng mode spells out the name of the streets 
contained in the directions. For example, the instructions 
"turn right on Elm" would also include the instruction "Elm 

5 is spelled E, L, M." By providing the spelling of street 
names, the spelling mode makes the streets easier to find and 
the route instructions easier for a user to follow. 

At step 118, saying "back up" or pressing the "7" key 
allows the user to back up to the previous step in the 

10 directions. 

At step 120, saying "repeat" or by pressing the "8" key 
cause the current direction step to be repeated. At step 122, 
the directions can be skipped ahead one step by requesting 
"skip ahead" or by pressing 9. At step 124, operator assis- 

15 tance can be obtained by requesting "operator" or by press- 
ing the "0** key. At step 126, the playback of the route 
directions can be paused and the available instructions menu 
can be played by requesting "instructions" or by pressing the 
star key * followed by zero. 

20 After each step of the route instructions, if no command 
is given within a timeout period, the system will continue 
automatically to the next route instruction step. Preferably, 
the timeout is configurable by the system but is typically set 
to a range of 2-4 seconds. In addition, each step of the route 

25 instructions may be proceeded by a step number identifying 
that step of the route directions (e.g., Step X: Turn <direc- 
tion> on <street>). 

After each step of the directions at step 128, a determi- 
nation is made whether the end of the route instructions or 

30 directions has been reached (i.e., the last step) or there are 
additional steps in the route instructions remaining to reach 
the destination. Preferably, the last step of the route instruc- 
tions will indicate total trip distance and need not be 
preceded by a step number. 

35 At step 130, if there are additional route instruction steps 
remaining, the system is in a post step state to play the 
remaining direction steps. At step 132, if the end of the route 
instructions have been reached, the system is in a post step 
state ready to proceed to the next step of the directions. 

40 Referring now to FIG. 7, shown are the commands 
available at step 130 of FIG. 5 after each step of the 
directions have been played. As shown in FIG. 7, the 
playback of the route instruction can be started over firom the 
beginning by stating "start over" or by pressing the "1" key 

45 on the keypad at step 140. At step 142, the user can say 
"continue" or press the "2" key to proceed to hear the next 
step of the route instructions. At step 144, the user can pause 
the playback of the directions by requesting "pause" or by 
pressing the "5" key. Playback can be resumed by requesting 

50 "resume" or by pressing the "5" key again. The menu 
selections available from the pause features are those pre- 
viously shown in FIG. 6. 

At step 146, the current step of the route instructions is 
repeated in a spelling mode. The spelling mode spells out the 

55 name of the streets in the route instructions as described at 
step 1X6 of FIG. 5. 

At step 148, saying "back up" or pressing the "7" key 
allows the user to back up to the previous step in the route 
instructions. 

60 At step 150, the current step of the route instructions can 
be repeated by saying "repeat*' or by pressing 8. At step 152, 
the directions can be skipped ahead one step by saying "skip 
ahead'* or by pressing 9. At step 154, operator assistance can 
be obtained by requesting "operator" or by pressing the "0*' 

65 key. At step 156, the playback can be paused and the 
available instructions menu can be played by requesting 
"instructions" or by pressing the star key * followed by zero. 
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At Steps 158 and 160, unrecognized speech commands interconnected computer readable media, which exist exclu- 

and invalid key entries are handled. FIG. 8, shows allowing sively on a processing system or is distributed among 

callers 3 chances to enter a valid selection. After an invalid multiple interconnected processing systems that may be 

selection at step 160, callers will be informed of their invalid local or remote to the processing system, 

selection and given an opportunity to select another option 5 The present embodiment preferably includes a software 

at step 162. At step 164, the number of consecutive invalid module as a set of computer executable software inslruc- 

selections are maintained. At step 166, after 3 invalid tions. The software instructions are executed as data bits by 

selections, users will be transferred to a live operator for the CPU with a computer memory system. The software 

assistance. Likewise, users will have 3 chances to speak a instructions cause CPU to access the gain lookup table 

valid command or be transferred to a live operator. lO preferably stored in data memory. In an alternate 

Preferably, the live operator will have access to the user's embodiment, the software instructions may evaluate a func- 

session including his requests for information and the tion to calculate the appropriate amplification gain without 

received responses. FIGS. 9 through 12 show an illustrative requiring an access to a gain lookup table stored in memory, 

embodiment of a menu command structure for handling The executable code may implement, for example, the 

unrecognized commands. is methods described in further detail below. 

RGS. 9-11 shows exemplary error handling for unrec- Referring to FIG. 13, the starting location, usually the 

ognized commands received from the user. The exemplary current location of the caller, can be determined at step 200 

embodiment of the error handling shown in FIG. 9 allows 3 using an ALI technology. The AU can identify the location 

consecutive unrecognized commands or errors before send- of the caller implementing any of the ALI technologies 

ing the user to a live operator 170 for assistance. After each 20 previously described. The- caller's location can then be used 

unrecognized command, the user may be given additional to populate the call detail, which is passed onto to the 

assistance as shown in FIGS. 10 and 11. operator console to be available for the operator during the 

FIG. 12 shows the menu commands at the post directions transaction with the caller, 
of step 132 of FIG. 5. After the route directions have been At step 202, the starting location determined in step 200 
played, the available commands are played back to the user 25 can be used with a selected destination location and a route 
at step 170. At step 172, requesting "start over" or entering between the two locations can be mapped. In an illustrative 
a "1" key starts the direction over at the beginning. At step embodiment, a database can provide the geographical loca- 
174, requesting "repeat'* or entering the "8" key repeats the tions of the starting and destination locations. The route is 
current step direction. At step 176, requesting '1?ack up" or mapped between the geographical starting and destination 
entering the "7" key goes back to the beginning of the 30 locations with a software program. Preferably, a route data- 
previous step and continues the playback of the route base is used to determine the route between the two starting 
directions. At step 178 the available instructions menu can and destination locations as previously described above. At 
be played by requesting "instructions" or by pressing the star step 204, the step-by-step street driving route instructions 
key * followed by zero. At step 180, operator assistance can are determined to provide specific directions from the star- 
be obtained by requesting "operator" or by pressing the "0" 35 ing location to the destination location. The route instruc- 
key. At steps 182 and 184, invalid entries and unrecognized tions can be determined according using a routing database 
commands are handled as previously described. as previously described. 

The present embodiment may also be realized as a method At step 206, the step-by-step route instructions are seg- 

of providing information over a telephone line. The method mented and demarcated between each step of the instruc- 

may be embodied in the form of executable code running on 40 tions. For example, at each point in the instructions where 

a processing system with a high-speed Central Processing the route instruction instructs the caller to changes from one 

Unit ("CPU") 48 and a memory system 52. In accordance road to another i.e., "turn right on ELM," "turn right on 

with the practices of persons skilled in the art of computer Main Street." The segmentation of the route instructions 

programming, the present invention is described below with provides convenient points for the interactive user interface 

reference to acts and symbolic representations of operations 45 to stop, pause, and repeat the driving instructions. Use of the 

that are performed by the computer processing system, demarcated route instructions to provide a convenient 

unless indicated otherwise. Such acts and operations are mechanism for the call to control the playback of route 

sometimes referred to as being "computer-executed", or driving instructions is described previously above. 

"CPU executed." At step 208, the interactive user interface communicates 

It will be appreciated that any symbolically represented 50 the demarcated route instructions. The interactive user inter- 
operations or acts described include the manipulation of face provides a convenient interface to allow the user to 
electrical signals by the CPU. The electrical system repre- receive route instructions over the telephone. The user 
sent data bits which cause a resulting transformation or interface preferably uses the demarcated route instructions 
reduction of the electrical signal representation, and the to allow a user to control the playback of the instructions as 
maintenance of data bits at memory locations in the memory 55 described in reference to FIGS. 4—12. 
system to thereby reconfigure or otherwise alter the CPU's The present embodiment preferably includes logic as a 
operation, as well as other processing of signals. The computer executable software instructions. The software can 
memory locations where data bits are maintained also be adapted to be executed by any general purpose Computer 
include physical locations that have particular electrical. Processing Unit ("CPU") or microprocessor controlling the 
magnetic, optical, or organic properties corresponding to the 60 operation of the system. Although the illustrative system 
stored data bits. uses an IBM RISC 6000 computer system may other com- 

The data bits may also be maintained on a computer puter systems/microprocessors are suitable for implement- 
readable medium including magnetic disks, optical disks, ing the described system and methods. The microprocessor 
organic disks, and any other volatile or (e.g.. Random executes software that can be programmed by those of skill 
Access memory ("RAM")) no n- volatile (e.g., Read Only 65 in the art to provide the desired functionality as described in 
Memory ("ROM")) storage system readable by the CPU. the included Figures. The software can be represent as a 
The computer readable medium includes cooperating or sequence of binary bits maintained on a computer readable 
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medium including magnetic disks, optical disks, organic 
disks, and any other volatile or (e.g., Random Access 
memory ("RAM")) non-volatile firmware (e.g., Read Only 
Memory ("ROM")) storage system readable by the CPU. 
The memory locations where data bits are maintained also 
include physical locations that have particular electrical, 
magnetic, optical, or organic properties corresponding to the 
stored data bits. The software instructions are executed as 
data bits by the CPU with a memory system causing a 
transformation of the electrical signal representation, and the 
maintenance of data bits at memory locations in the memory 
system to thereby reconfigure or otherwise alter the system's 
operation. 

The illustrative embodiment improves the usabiUty of the 
system and improves the productivity of the operator con- 
sole position. The automatic location identification a elimi- 
nates the need for callers to identify their location when 
calUng into the system for assistance. The automatic location 
identification also speeds the handling of calls at the opera- 
tor console by eliminating the step of querying the caller for 
his location. The interactive user interface allows the user to 
more easily obtain the route instructions by providing a 
mechanism to start and stop the directions at convenient 
points. The interactive user interface also improves the 
productivity of the operator console by allowing the operator 
to pass the caller off to the interactive user interface and 
move on to handle the next caller. 

It is to be understood that both the foregoing general 
description and the following detailed description are exem- 
plary and explanatory and are intended to provide further 
explanation of the invention as claimed. Numerous modifi- 
cations and variations are possible. For example, the steps of 
the flow diagrams may be taken in sequences other than 
those described and the invention may be practiced with 
more or fewer elements than those shown. It is intended that 
the foregoing detailed description be regarded as illustrative 
rather than limiting. It is the following claims, including all 
equivalents, which are intended to define the scope of this 
invention. 

We claim: 

1. A system for providing a telephone caller driving route 
instructions from a calling location to a destination location 
comprising: 

an automatic location identification for automatically 
identifying the calling location from where the tele- 
phone caller originated the telephone call; 

a database for mapping route instructions from the calling 
location to the destination location, wherein the route 
instructions are coded at demarcated points to allow for 
manipulation of the route instructions during playback; 
and 

an interactive user interface accessible over a telephone 
call for communicating route instructions to the tele- 
phone caller, wherein the interactive user interface 
provides a user with commands at each demarcated 
point to manipulate, during playing, the playback of the 
route instructions. 

2. The system of claim 1 wherein the automatic location 
identification for identifying the calling location further 
comprises: 

automatic number identification for identifying the calling 

party telephone number; and 
a telephone number database for determining the address 

of the calling party telephone number. 

3. The system of claim 2 wherein the telephone number 
database comprises a name, address and telephone number 
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database (NATO) containing geo-code information identi- 
fying the geographic location of the telephone caller. 

4. The system of claim 3 wherein the geo-code informa- 
tion identifying the geographical location comprises longi- 
tude and latitude coordinates. 

5. The system of claim 1 wherein the route instructions 
comprise street driving instmctions between the calling 
location and the destination location. 

6. The system of claim 1 wherein the route instructions are 
slcp-by-step driving instructions which have been demar- 
cated at each step of the instructions. 

7. The system of claim 1 wherein the automatic location 
identification identifies the caller location comprising a 
dedicated radio-location network. 

8. The system of claim 1 wherein the automatic location 
identification identifies the caller location comprising a GPS 
location system. 

9. The system of claim 1 the interactive user interface 
comprising: 

a gateway for translating the route instructions to an audio 

box for speech synthesis; and 
an audio box for receiving the translated route instructions 

and verbalizing the route instructions. 

10. A system for providing a telephone caller driving 
instructions from a calling location to a destination location 
comprising: 

an automatic location identification for automatically 
identifying the calling location from where the tele- 
phone caller originated tfie telephone call, comprising: 
automatic number identification for identifying the 
calling party telephone number; and 
a telephone number database for determining the 

address of the calhng party telephone number; 
a database for mapping route instructions from the 
calling location to the destination location, 
wherein the route instruaions are coded at demar- 
cated points to allow for manipulation of the route 
instructions during playback; and 
an interactive user interface accessible over a telephone 
call for communicating route instructions to the 
telephone caller, wherein the interactive user inter- 
face provides a user commands at each demarcated 
point to manipulate, during playing, the playback of 
the route instructions, comprising: 
a gateway for translating the route instructions to an 

audio box for speech synthesis; and 
an audio box for receiving the translated route 
instructions and verbalizing the route instructions, 

11. A method of providing driving instructions between a 
starting location and destination location to a telephone 
caller comprising the steps of: 

locating the starting location using an automatic location 
identification technology; 

mapping a driving route between the starting location and 
the destination location; 

determining step-by -step driving instructions in accor- 
dance with the step of mapping the driving route; 

coding the driving instructions at each step of the direc- 
tions into segments, wherein the coded driving instruc- 
tions can be manipulated during playback by com- 
mands provided to a user at each segment using an 
interactive user interface; and 

communicating the driving instructions over a telephone 
line with an interactive user interface adapted to com- 
municate the coded driving instructions. 

12. The method of claim 11 wherein the interactive user 
interface comprises the capability of pausing the instructions 
at a coded segment. 
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13. The method of claim 11 wherein the interactive user 
interface comprises the capability of repeating the segment 
of the instructions at a coded segment. 

14. The method of claim 11 wherein the interactive user 
interface comprises the capabUily of repeating the instruc- 5 
tions from the beginning at a coded segment. 

15. A system for providing a telephone caller driving route 
instructions from a starting location to a destination location 
comprising: 

a database for mapping route instructions from the start- 10 
ing location to the destination location, wherein the 
route instructions arc coded at demarcated points to 
allow for manipulation of the route instructions during 
playback; and 

an interactive user interface accessible over a telephone 
call for communicating route instructions to the tele- 
phone caller, wherein the interactive user interface 
provides a user with commands at each demarcated 
point to manipulate, during playing, the playback of the 
route instruction. 

16. The system of claim 15 wherein the commands are 
activated by a telephone caller's voice. 

17. The system of claim 15 wherein the commands are 
activated by a depressing a key from the DTMF keypad. 

18. The system of claim 15 wherein the route instructions 
comprise street driving instructions between the starting 
location and the destination location. 
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19. The system of claim 15 wherein the route instructions 
are step-by-step driving instructions which have been 
demarcated at each step of the instructions. 

20. The system of claim 15 the interactive user interface 
comprising: 

a gateway for translating the route instructions; and 
an audio box for receiving the translated route instructions 
and verbalizing the route instructions. 

21. A system for providing a telephone caller driving route 
instructions from a starting location to a destination location 
comprising: 

a database for mapping route instructions from the start- 
ing location to the destination location, wherein the 
route instructions are coded at demarcated points to 
allow for manipulation of the route instructions during 
playback; 

an interactive user interface accessible over a telephone 
call for communicating route instructions to the tele- 
phone caller, wherein the interactive user interface 
provides a user with commands at each demarcation 
point to manipulate, during playing, the playback of the 
route instructions; 

a gateway for translating the route instmctions to an audio 
box for speech synthesis; and 

an audio box for receiving the translated route instructions 
and verbalizing the route instructions. 

♦ ♦ * ♦ * 
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[57] ABSTRACT 

An automated interactive record system for automati- 
cally indexing a set of data obtained from a number of 
internal and external input sources, A set of data is 
organized along at least one indexing dimension. Spe- 
cific data events are identified from the set of data ac- 
cording to predetermined methods. An index of data 
objects associated with the specific data events is then 
established. The index of data objects is then displayed 
along the indexing dimension and actuators are defined 
for manipulating the data objects that are associated 
with specific data events. 
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live 10 a point or group of points, that constitute a spe- 
AUIOMATED INTERACTIVE RECORD SYSTEM cific event as predefined by the system. 

It is a further object of this invention to provide a 
FIELD OF INVENTION system which reduces viewer confusion and disorienta- 

This invention relates to an automated interactive ^ ^y maintaining a continual display of the data 
recover svstem. and more panicuiarlv to a system that shown on the axis while input or accessed data objects 
displays data along at least one indexing dimension and be viewed mdcpendently with reference to the 

allows ready access to associated data relative to some ^'s. 

point or set of points along the dimension and the input It is a further object of this mvention to provide a 
of data 10 be displayed at given locations upon the di- system which indexes data from preexistmg data bases 
mension. otherwise input data. 

It is a further object of this invention to provide a 
BACKGROUND OF INVENTION system which allows the input of data from a variety of 

In the medical care field, data on a patient's recovery both manually, internally generated, and automatically 
or condition is often entered by hand after performing a activated sources. 

therapeutic function or observing various monitors and It is a further object of this invention to provide a 
measuring devices. This data is combined into a series of system allowing input and display of data objects in a 
data collections often without easy access to any single variety of manners and representations, including lex- 
piece of data or a method of correlating data from dif- tual, audio, graphical and pictorial, 
ferenl places within the collection as a whole. This lack jj g fiirther object of this invention to provide a 
of organization and easy access makes it difficult for system allowing the user direct screen interaction to 
health care personnel to quickly identify trends and key manipulate data. 

events in the patient's condition. Further, the prospec- js a further object of this invention to provide a 

tive administration of procedures, medication or tests ^ system which provides access to and input of data in a 
may also prove troublesome since this type data is not 25 ^^^^^^^ highly effective in the fields of prcwress control, 
generally combined or correlated with already existing medical care and other areas requiring organization of 
monitor data that is gathered on the patient. This mom- various forms of data over a dimension, 
tor data, however, could in fact be a necessary precon- j^-^ invention results from the realization that a truly 
dition to performing thii prospective admmisiration of ^^^^xWc data managemem and control system may be 
c^*"^- . , . J achieved through the combination of the two powerful 

Even when health care personnel store patient data independent concepts of hypermedia and indexing of 
on a computer, n is not adequately organized and pres- ^^^^ ^^^^ dimensions, and through a sys- 

ented with respect to the overall process of monitoring ^^^^ ^^^^.^^^ . ^^^^ ^^^^ ^ ^^^^^^^ 

condition or recovery^ which ,s often represented by^^ ^^^^^^^ ^^^^ .^^^^^^^^ associates these data 
stnp chart recordmg of a certain vital sign. Hence 35 ^.^^ ^ significance along the 

correlating ^^^er d^^^ a^^^^^^^ indexing dimension to^ive the viewer a clear, uncon- 

stnp chart recording often P/^^^^^^^^^^ fusing, display with easy access to. and control of, data, 

mg this set of da am order »^ '^^f^^^^^^^^^^^ This invention features an automated interactive re- 

once It IS correlated with similar data taken at earner • i j 

intervals, proves even more troubling. In specific areas 40 cord system. This system includes means for organizing 
of medical care, technology is now available to monitor ^ set of data along at least one indexing dimension^ 
several parameters of the subject's condition through a There are means provided for identifying specific data 
computer. However, the problem with many computer events from the set of data according to predetermined 
monitoring systems is the continuing inability of the ""ethods. There are means for establishing an index of 
user to correlate the many layers of interconnected data 45 data objects associated with the specific data events, 
in a manner in which the initial broad perspective of the There are further means for displaying this index of data 
data is not lost. This is the problem encountered with objects along the indexing dimension and means for 
hypermedia in which, as progressively smaller conccn- defining actuators for manipulating the data objects 
traiions of data are viewed, the large picture becomes indexed with specific data events, 
lost, thus leading to viewer disorientation and confu- 50 In a preferred embodiment, the means for establishing 
sion. A system is required to retain perspective with the an index of data objects may further include marker 
larger view of data while concurrently allowing Ihc means for individually marking data events. The marker 
viewer to delve into the layers of hypermedia. A need means may further include means for designating the 
thus arises in medicine as well as in many other areas of location of data associated with specific data events, 
process control and monitoring of nonmedical systems, 55 means for specifying the configuration of actuators, 
to devise a method to track and correlate various data means for determining a symbolic representation to be 
from a variety of sources, including the computer's own displayed for a given data object, and means for desig- 
internal calculations with reference to a time line or nating the operation to be effected for a specific data 
similar dimension that retains a larger and lasting per- event. The means for defining actuators may further 
spcctivc. ^ include means for bounding interactive areas on the 

display screen corresponding to locations along the 
SUMMARY OF IlvTVENTION indexing dimension, and those means for bounding may 

It is therefore an object of this invention to provide a include means for operating the actuators in response to 
system for the automatic collection, organizing and a touching of the interactive areas and in response to a 
indexing of a set of acquired signal data, or other previ- 65 contacting of the interactive areas with a cursor. The 
ously stored data, and display of this index data graphi- means for displaying the index of data objects and spc- 
cally on one or more axes where the viewer maintains cific data events may further include means for gcncrat- 
the ability to access or input specific data objects, rela- ing descriptive pictorial icons which are symbolic rcp- 
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resentations of data objects and specific data events and 
which are associated with specific data events along the 
indexing dimension. The means for organizing a set of 
data may further include an indexing dimension, which 
is lime. The means for identifying specific data events 
may include means for comparing the set of data lo a 
reference and means for reviewing data objects lo gen- 
erate additional sp)ecific data events. The means for 
organizing this set of data may include means for graph- 
ically ploiling the set of data on a display screen. This 
means for graphically plotting may further include 
means for graphically displaying the minimum, maxi- 
mum and mean values of data over a given interval. 
Finally, the means for establishing an index of data 
objects may include a data object comprising at least 
one of a plurality of data fields including audio, visual, 
textual, numerical, algorithmic and graphical data. 

DISCLOSURE OF PREFERRED EMBODIMENT 

Other objects, features and advantages will occur to 20 
those skilled in the art from the following description of 
preferred embodiment and the accompanying drawings 
in which: 

FIG. 1 is a fiow chan of an interactive record system 
according lo this invention: 25 

FIG. 2 is a more detailed view of the flow chart of 
FIG. 1: 

FIG, 3 is a display screen for the system generated by 
the system according to this invention: 

FIG. 4 is a block diagram of an interactive record 30 
system according to this invention for medical applica- 
tions showing various input, output and peripheral de- 
vices: 

FIG. 5 is a flow chart for a specific medical applica- 
tion of the interactive record system according to this 35 
invention for indexing blood pressure data: 

FIG. 6 is a more detailed view of the flow chart of 
FIG. 5 showing the formation of specific data events; 

FIG. 7 is a flow chart showing the acquisition of an 
image by the system; 

FIG. 8 is a flow chart showing the retrieval of an 
image through activation of a pictorial icon: 

FIG. 9 is a flow chart showing the retrieval of a high 
fidelity blood pressure waveform data object through 
activation of an actuator: and 

FIG. 10 is a flow chart showing the continuing opera- 
tion of the system. 

In FIG. I there is shown a flow chart of an interac- 
tive record system according to this invention in which 
a set of raw data 10 is organized along an indexing 50 
dimension 12. The indexing dimension may be lime or a 
similar dimension that may be used to organize and 
correlate data from a variety of sources. An indexing 
dimension is a variable, parameter, or characteristic 
intrinsic to the data which is being accumulated, that is 55 
selected because of its commonality. Other examples of 
such variables or parameters include sex or age of hospi- 
tal patients; day or month of the year; or dimensions 
such as length. By means of this commonality, data 



40 



45 



mension, specific data events 14 are created through 
distinguishing sets of data into given intervals. Data 
objects 18 from other sources are associated 16 with 
specific data events according to predetermined crite- 
ria. The data objects and data events are then displayed 
20 relative to the indexing dimension as symbols. These 
symbols are located in relation to the associated specific 
data events allowing the viewer to clearly see the rela- 
tionship between the data objects and indexed data 
events. Once the data objects are displayed along the 
indexing dimension, actuators are created 22 to allow 
the data objects to be manipulated. The actuator is an 
interactive area, generally placed around the symbol of 
a data object or data event, that allows the viewer to 
activate the underlying data contained in the data object 
or referenced by the data events. The data object may 
be one of a variety of multimedia collections of data 
including pictorial, graphical, audio, textual or numeri- 
cal data. The data object or data event as displayed may 
also prompt the viewer for further interaction and in- 
put. 

The interactive record system of FIG. 1 is shown in 
a more detailed flow chart in FIG. 2 in which the cre- 
ation of data events 14 from an index of data along a 
dimension is accomplished through the comparison 26 
of the indexed data with reference data 24. Data objects 
18 are then associated 16 with data events. This is acr 
complished by marking 28 each data event to distin- 
guish it from others and make it uniquely identifiable. 
Then, a location is designated 30 for each data event 
along the indexing dimension. The configuration of the 
actuators for each data event is then specified 32. The 
symbol to be displayed, if any. in relation to a given data 
event for a data object is then determined 34. Finally, 
the operation to be performed for each data object is 
determined 36. The operation performed for a data 
object may include the creation of additional data 
events 19. Data objects are reviewed and algorithmic 
data is retrieved and acted upon. These algorithms may 
specify new criteria or locations for data events which 
will then be noted by the system. Data objects and data 
events are then displayed 20 as described in FIG. 1 and 
actuators are defined 22, Defining of actuators is ac- 
complished by determining the boundaries of active 
areas on the display screen 38 in which a given form of 
contact will cause an operation, and then by defining 
the exact function to be performed 40 for each of these 
active areas upon activating them. 

FIG. 3 shows a specific example of a display screen 
for the system disclosed. An indexing dimension display 
70 is shown along the top of the screen The upper half 
of this indexing dimension contains vertical line plots of 
data trends 60 which represent the maximum 61, mini- 
mum 63 and mean 65 values of data sampled over a 
given interval. This interval may represent the time of 
input. The dimension In this example plots continually 
from left to right. 

The bottom half of the dimension display 70 conuins 
pictorial icons 50 representative of data objects and data 



from a variety of sources can be organized and corre- 60 events and placed below and relative to data events 



lated. 

Since the interactive record system according to this 
invention can be used in a number of applications, an 
indexing dimension(s) is selected for a specific applica- 
tion and the system is configured to use that indexing 65 
dimension(s). In the case where time is the chosen in- 
dexing dimension, the raw data is organised and corre- 
lated in chronological fashion. From this indexing di- 



with which the system has associated them. A data 
event in this example is represented by a grouping 64 of 
venical line trends 60 that are all valued within a certain 
predetermined threshold level. In this example, group- 
ings appearing beyond predetermined threshold levels 
form one basis for the creation of new data events. 

A dotted line 66 appears around the trends compris- 
ing a data event. This dotted line is a representation of 
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an actuator. This actuator is an unseen interactive area then continues by returning to the next sample 120. 

which, when coniacied either by physical or cursor Finally, if the system determines that the current mean 

touch, will operate to display predetermined associated is above the threshold 110 and the previous interval's 

data objects. Similar active areas 54 exist for each of the mean was also above the threshold 112 then the system 

pictorial icons 50. which, when contacted, carry out a 5 finds no event and simply continues 120 to the next 

predetermined function. sample. ^ • ^ • j- 

The bottom half of the display screen 52 carries a Flow chart 127. FIG. 6. updates the mdexmg dimen- 

work display window, in which the contents of given sion display and actuator location. When the system 

data objects' may be viewed or manipulated. In this determines the beginning of a new data sampling mter- 

particular example a pictorial data object is revealed. 10 val 126 it calculates the horizontal location of the dis- 

The other part (right halO of bottom half of the dis- play screen interval using the interval's dimensional 

play screen contains a series of "buttons" which in this value, in this example its time value 128. The system 

particular example allow input of an image 44. a wave- then determines whether the display screen pixel is at 

form 46 and voice 48 to form data objects. Means for the same location as that for the previous interval 130 

other types of input may also be provided. 15 since several intervals may be condensed into one pixel. 

Block diagram 72. FIG. 4, discloses a specific em- If it is not the same pixel, then a new Ime segment from 

bodiment of the system in which a computer 74 for high to low with a highlighted mean value is plotted 

operating the system is provided. This computer may be 136. If the pixel, however, is the same as that m the 

an Apple Macintosh model. Data is stored on and re- previous interval, then the system combmes the previ- 

irieved from a disk unit 78. A touch-sensitive interactive 20 ous and current interval and calculates a new combmed 

screen 76 mav be provided for direct, physical, user high, low and mean value 132 which it replots 134 

interaction with the data displayed. A mouse unit 82 is within that pixel space if necessar>'. Given either a new 

also provided in order to position a cursor on the inter- pixel or not. the system then determines whether there 

active screen. »s a new event 138 within the interval from the notation 

A keyboard 84 is provided for data entry and manipu- 25 of events as described in FIG. 5. If no new event has 

lation. in this embodiment medical information on drug occurred, the actuator surrounding the plotted trends is 

98, urine 99 and blood 100 data is input via this key- continued to include the current pixel 140. The system 

l^3j.jj then continues 144 through the routine again upon a 

Further data, such as lab reports 81. may be acces.sed new interval. If, however, the system determines that a 

via a network 80 linking other data bases. 30 new event has been noted in this interval it begins a new 

Data may also be input to the svstem to form data actuator with the present pixel 142. This new actuator 

objects through a number of peripheral devices. Visual activates any data objects associated with the new 

images may be input via an imaging device 90 which, event. At this point the system cominues 144 to the next 

through use of a frame grabber 91. creates a digitized interval. 

image for storage as data in the system. Voice and simi- 35 Flow chart 145, FIG. 7, enters data by viewer activa- 

lar audio inputs may be stored as data through use of a tion of a button. In this example an image taking button 

microphone 88 and voice acquisition unit 94. Various 44 as depicted in FIG. 3 is activated 146. The activation 

monitoring signals mav also be stored by connecting the of this button instructs the system to activate the frame 

monitor 86, which in this example reads blood pressure grabber 148 which digitizes an instantaneous image 

or electrocardiogram, to a signal acquisition unit 96. 40 input by an imaging device such as a video camera. This 

Other forms of data mav further be input using the image data is then written onto a disk for storage 150. 

proper devices and systems to create data readable by The image data is marked as an event with appropriate 

the svMcm. caiegorizaiions 152, in this example, time of input, file 

Flow chart 103, FIG. 5. constructs blood pressure name and the prevailing conditions. The image is then 

trend data arrays in a specific medical example. A sam- 45 displayed by the system in the work window 42 as 

pie rate of one sampling each l/250th of a second 102 is depicted in FIG. 3. A pictorial icon of the image is then 

chosen in which the blood pressure signal from a moni- created and displayed 156 relative to the indexing di- 

tor is read 104. The ongoing high, low and mean values mension. in this example time of creation. Such an icon 

for each data sampling interval (in this example, each . 50 is depicted in FIG. 3. An actuator 54 is then created 

five seconds) is calculated 106 from the signal each time 50 158, as depicted in FIG. 3. around the pictorial icon to 

a new sample is read. A condensed waveform of this allow future access of the data of the image. The system 

signal is then updated 108 with the new piece of data. then continues 160 awaiting the next input instruction. 

The system then determines if the latest mean value for Flow chart 161, FIG. 8, describes the retrieval of data 

the interval is above a predetermined threshold 110. If through the operation of an actuator placed around a 

not, the system then determines whether the stored 55 pictorial icon. An actuator placed around a pictonal 

mean for the previous interval was below the threshold icon is activated 162 by interacting with the screen 

122. If the previous interval was not below the thresh- either by a physical touch of the active area or through 

old then the current interval is noted as an event from use of a cursor. Activation of the icon serves to retrieve 

high to low 124. The system then continues by return- the data object into system memory 164, in this example 

ing to the next data sample and repeating the routine 60 a stored image. The system then displays this image in 

120. If the mean of the current interval is found to be the work window 166. 

above the threshold 110. then the system will determine Flow chart 169, FIG. 9, similar to the method for 

whether it was above the threshold in the previous retrieving data from an icon, shows the retrieval of 

interval 112. If not. then the system notes an event of trend event related data through the activation of a 

low to high value 114. The system then triggers a high 65 trend event located actuator 66 as depicted in FIG. 3. 

fidelity blood pressure waveform event 116 in which An actuator located around a group of trends on the 

fifteen seconds of raw blood pressure monitor data is indexing dimension comprising a trend event is opcr- 

taken by the system to be stored on a disk. The system ated 168 by user interaction with the display screen, 
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either ihrough physical louch or cursor contaci. This 
aciuamr activation causes the system to retrieve a re- 
lated data object 170. in this example high-fidelity 
waveform data. This waveform is then displayed in the 
work window 172. 

Finally, flow chart 173. FIG. 10. describes the con- 
tinuing operation of the system. The system will either 
review previous collected data or continue building in 
new data 174. An initial dictionary of new data is in- 



8. The interactive record system of claim 7 in which 
said means for bounding includes means for operating 
said actuators in response to a touching of said interac- 
tive areas. 

9. The interactive record system of claim 7 in which 
said means for bounding includes means for operating 
said actuators in response to a conlaciing of said interac- 
tive areas with a cursor. 

10. The interactive record system of claim 1 in which 



dcxcd into a primary index of trend data events 176. 10 said means for displaying includes means for generating 



These trend data events then form the basis for the 
creation of trend data objects. The data objects are. in 
turn, reviewed to build a secondary index of data events 
based upon the information present in the data objects 
178. in this example, blood pressure waveform data 15 
objects. Trend events are then displayed by the system 
using vertical line plots 180. Actuators are then created 
for each trend event to retrieve associated data objects 
182. All primary events, including those generated from 



descriptive pictorial icons which are symbolic represen- 
tations of said data objects and said specific data events 
and which arc associated with said specific data events 
along said indexing dimension. 

11. The interactive record system of claim 1 in which 
said means for organizing includes an indexing dimen- 
sion which is time. 

12. The interactive record system of claim 1 in which 
said means for identifying specific data events includes 



data objects, are then displayed using predetermined 20 ^^^^^ comparing said set of data to a reference. 



pictorial icons which describe the type of underlying 
events 184. Actuators are then created to activate these 
pictorial icons in order to retrieve the underlying data 
objects represented by the icons. Finally, the system 
returns to repeat this process, 25 

Although specific features of the invention are shown 
in some drawings and not others, this is for convenience 
only as each feature may be combined with any or all of 
the other features in accordance with the invention. 

Other embodiments will occur to those skilled in the 30 
an and are within the following claims: 

What is claimed is; 

1. An imeraciive record system for automatically 



13. The interactive record system of claim 1 in which 
said means for identifying specific data events includes 
means for reviewing data objects to generate additional 
specific data events. 

14. The interactive record system of claim 1 in which 
said means for organizing a set of data includes means 
for graphically plotting said set of data on a display 
screen. 

15. The interactive record system of claim 14 in 
which said means for graphically plotting includes 
means for graphically displaying minimum, maximum 
and mean values of data over a given interval. 

16. The interactive record system of claim 1 in which 



orcanizme and mdexing data obtamed from a number of _ ui- l - - i j -i «u; * 

. I ^. , . , „ „ „ said means for establishing includes a data object com 

miernal and external mput sources, comprising: 35 . , ^ , . 

means for organizing a set of data along at least one 

indexing dimension; 

means for identifying specific data events from said 

set of data according to predetermined methods; 



prising at least one of a plurality of data fields including 
audio, visual, textual, numerical, algorithmic and graph- 
ical data. 

17. An interactive record system for automatically 



means for establishing an index of data objects associ- 40 organizing and indexing data obtained from a number of 
aied with said specific data events: >"»ernal and external input sources, compnsing: 



means for displaying said index of data objects and 
specific data events along said indexing dimension; 
and 

means for defining actuators for manipulating said 45 
data objects indexed with said specific data events. 

2. The interactive record system of claim 1 in which 
said means for establishing an index of data objects 
includes marker means for individually marking said 
specific data events. 50 

3. The interactive record system of claim 2 in which 
said marker means includes means for designating the 
location of data associated with said specific data 
events. 

4. The interactive record system of claim 2 in which 55 
said marker means includes means for specifying the 
configuration of said actuators. 

5. The interactive record system of claim 2 in which 
said marker means includes means for determining a 



means for organizing a set of data relative to at least 

one indexing dimension: 
means for identifying specific data events from said 

set of data according to predetermined methods; 
means for plotting said specific data events on a dis- 
play screen along an axis representative of said 
indexing dimension; 
means for establishing an index of data objects from 
internal sources and external sources associated 
with said specific data events; 
means for displaying said index of data objects and 
said specific data events relative to said specific 
data events; and 
means for defining actuators associated with said 
specific data events for manipulating said data ob- 
jects. 

18. The interactive record system of claim 17 in 
which said means for establishing an index of data ob- 



symbolic representation to be displayed for said data 60 jects includes means for individually marking said spe- 
objccts. cific data events. 

6. The interactive record system of claim 2 in which 19. The interactive record system of claim 18 in 
said marker means includes means for designating the which said marker means includes means for designat- 
operation to be effected for a specific data event. ing the location of data associated with said specific 

7. The interactive record system of claim 1 in which 65 data events. 

said means for defining actuators include means for 20. The interactive record system of claim 18 in 
bounding interactive areas on a display screen corre- which said marker means includes means for specifying 
sponding to locations along said indexing dimension. the configuration of said actuators. 
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21. The inieractive record system of claim 18 in 
which said marker means includes means for determin- 
ing a symbolic representation to be displayed for a data 
object. 

22. The interactive record system of claim 18 in ^ 
which said marker means includes means for designat- 
ing the operation to be effected for a specific data event. 

23. The interactive record system of claim 17 in 
which said means for identifying specific data events 
includes means for reviewing data objects to generate 
additional specific data events. 

24. The interactive record system of claim 17 in 
which said means for defining actuators includes means 
for bounding interactive areas on a display screen corre- j ^ 
sponding to locations along said axis. 

25. The inieractive record system of claim 24 in 
which said means for bounding includes means for oper- 
ating said actuators in response to a touching of said 
interactive areas. 20 

26. The interactive record system of claim 24 in 
which said means for bounding includes means for oper- 
ating said actuators in response to a touching of said 
interactive areas. 

27. The interactive record system of claim 17 in 25 
which said means for displaying includes means for 
generating descriptive pictorial icons which are sym- 
bolic representations of said data objects and said spe- 
cific data events and which are as.sociaied with said 
specific data events along said indexing dimension. ^0 

28. The inieractive record system of claim 17 in 
which said means for establishing includes a data object 
comprising at least one of a plurality of data fields in- 
cluding audio» visual, textual, numerical, algorithmic 
and graphical data. 

29. The interactive record system of claim 17 in 
which said means for displaying includes display con- 
trol means for representing data on said axis in magni- 
fied and reduced format. ^ 

30. A medical inieractive record system for automati- 
cally organizing and indexing data obtained from a 
number of patient monitoring source comprising: 

means for organizing a set of monitored data relative 
to time; 45 

means for determining desired values of data taken 
over a given interval of time; 

means for representing said values on a display screen 
as linear trends relative to a point displayed on an 
aMS representing time; 50 

means for identifying specific data events from said 
trends that are greater than or less than predeter- 
mined threshold values; 

means for establishing an index of data objects from 
automatic inputs, from internally generated data 55 
and from data input manually associated with said 
specific data events; 



means for displaying said index of data objects associ- 
ated with said specific data events using pictorial 
icons that symbolically represent said data objects: 

means for defining actuators associated with said 
specific data events for manipulating said data ob- 
jects: 

means for operating said actuators in response to a 

touching by at least one of a cursor touch and a 

physical touch; and 
means for accessing said data objects in response to a 

touching by at least one of a cursor touch and a 

physical touch. 

31. The medical interactive record system of claim 30 
in which said means for establishing includes a data 
object comprising at least one of a plurality of data 
fields including audio, visual, textual, numerical, algo- 
rithmic and graphical data. 

32. The medical interactive record system of claim 30 
in which said^means for establishing includes means for 
signifying the type of data objects to be created for said 
specific data events. 

33. The medical interactive record system of claim 30 
in which said means for displaying includes display 
control means for representing data on said axis in mag- 
nified and reduced format. 

34. A medical interactive record system for use in a 
cardiac catheterization laboratory for automatically 
organizing and indexing data obtained from a physio- 
logical monitor, comprising: 

means for organizing a set of physiological data rela- 
tive to time of input: 

means for determining the high, low and means val- 
ues taken over a given interval of time; 

means for graphing said values on a display screen as 
venical line trends relative to a point displayed on 
a horizontal axis representing time of input: 

means for identifying specific physiological events 
from said trends that cross a predetermined thresh- 
old value; 

means for establishing an index of data objects from 
physiological monitor data, iniernally generated 
data, automatically input data and data input by 
personnel, associated with said specific physiolog- 
ical events; 

means for displaying said index of data objects rela- 
tive to said specific physiological events using pic- 
torial icons that symbolically represent isaid data 
objects; 

means for creating actuators associated with said 

trends in order to manipulated said data objects; 
means for operating said actuators in response to a 

touching by at least one of a cursor touch and a 

physical touch; and 
means for accessing said data objects in response to a 

touching by at least one of a cursor touch and a 

physical touch. 
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[57] ABSTRACT 

Disclosed is a computer audio interface adapted to en- 
able blind or visually impaired users to locate icons 
positioned in rows in the background of a graphical user 
interface. Whenever the pointer of the interface is posi- 
tioned on the background, the system produces a dis- 
tinctive tone. Whenever the pointer is positioned in an 
icon row that is occupied by one or more icons, the 
system produces a distinctive chord. Whenever the 
pointer is positioned in the background and not in an 
occupied row, the system produces stereo effects that 
give the user mformation as to the relative left/right 
position of the pointer. Whenever the pointer is posi- 
tioned in an occupied row, the amplitude of the left and 
right stereo channels is controlled such that whenever 
the pointer is to the left of the leftmost icon, substan- 
tially all of the volume is produced by the left speaker 
and whenever the pointer is to the right of the rightmost 
icon, substantially all of the the volume is produced by 
the right speaker. Whenever the pointer is located be- 
tween the leftmost and rightmost icons, the volume is 
balanced substantially equally between the two speak- 
ers. 

19 Qaims, 3 Drawing Sheets 
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METHOD AND SYSTEM FOR ENABUNG A 
BLIND COMPUTER USER TO LOCATE ICONS IN 
A GRAPHICAL USER INTERFACE 

5 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to a system 
and method for enabling a blind or visually impaired 
user to use a graphical user interface, and more particu- 
larly, to a system and method for enabling a blind or 
visually impaired user to locate icons in a graphical user 
interface. 

2. Description of the Prior Art 

In recent years, there has been a move among com- 
puter application software developers toward graphical 
user interfaces (GUIS). In graphical user interfaces, 
objects are presented for users to manipulate in ways 
that are similar to the way that they are manipulated in 
the real work place. Objects, such as file cabinets, fold- 
ers, documents, and printers, are displayed on the 
screen as icons. Users manipulate these objects with a 
mouse to perform desired operations. For example, to 
file a document in a folder thai is located in a file cabinet 
in the real work place, the user opens the file cabinet, 25 
locates and opens the correct folder, and puts the docu- 
ment inside. In the electronic work place of the graphi- 
cal user interface, the user performs a similar process. 
The user opens the file cabinet icon, locates the correct 
folder icon, and drops the document icon in the folder. 30 
Because this is an electronic environment, users do not 
have to open the folder to put the document in it. How- 
ever, users have been able to use their knowledge of a 
real work place to perform this operation. 

Normally sighted persons find graphical user inter- 35 
faces intuitive and easy to work with. However, except 
for an occasional "beep" or "bong", graphical user 
interfaces are virtually silent and the vast majority of 
the information they provide to the user is visual. Thus, 
graphical user interfaces are essentially not usable by 40 
blind or severely visually impaired people. 

Blind and visually impaired computer users now ben- 
efit from many forms of adaptive technology, including 
speech synthesis, large-print processing, braille desktop 
publishing, and voice recognition. However, presently, 45 
almost none of the foregoing tools is adapted for use 
with graphical user interfaces. It has been suggested 
that programmers could write software with built-in 
voice labels for icons, Lazzaro, Windows of Vulnerabil- 
ity, Byte Magazine, June, 1991 page 416. Various syn- 50 
thetic or recorded speech solutions for making com- 
puter display screen contents available to blind persons 
have been suggested, for example in Golding, et al., 
IBM Technical Disclosure Bulletin, Vol. 26, No. 106, 
pages 5633-3636 (March 1984). and Bamett. et. al., IBM 55 
Technical Disclosure Bulletin, Vol. 26, No. lOA, pages 
4950-4951 (March 1984). Recently, there has been dis- 
closed a prototype of a system called IBM Screen Rea- 
der/PM. Schwerdtfeger. Making the GUI Talk, Byte 
Magazine, December 1991, page 118. According to the 60 
Schwerdtfeger article, a user of the IBM Screen Rea- 
der/PM system can maneuver a mouse over the display 
and use the keyboard or a separate keypad, and a voice 
synthesizer will describe an icon the GUI has displayed 
or the graphical text shown on the screen. Additionally, 65 
there have been suggested systems that include a mouse 
with a braille transducer so that a blind user may read 
text and obtain certain tactile position feedback from 



the mouse. Comerford, IBM Technical Disclosure Bul- 
letin Vol. 28, No. 3, page 1343 (August 1985), Affmito, 
et. al.. IBM Technical Disclosure Bulletin Vol. 31, No. 
12, page 386 (May 1989). However, while announcing 
various text items, either audibly or by means of a 
braille transducer in the mouse, may provide some in- 
formation to blind user, it does not enable the user to 
navigate about and locate objects on the computer dis- 
play screen. 

There has been suggested an audible cursor position- 
ing and pixel (picture element) status identification 
mechanism to help a user of an interactive computer 
graphics system locate data by using aural feedback to 
enhance visual feedback. As the cursor is stepped across 
the screen, an audible click is generated that varies in 
tone corresponding to the current sutus of each pixel 
encountered. With this combination in audible and vi- 
sual cursor feedback, it becomes a simple task to iden- 
tify the desired line by noting the change in tone as the 
cursor moves. For color display applications, each 
color is represented by a distinct tone so any single pixel 
may be distinguished from the surrounding pixels of a 
different color. Although the technique was originally 
developed for computer aided drafting, it has been sug- 
gested that this system is especially helpful for visually 
impaired or learning disabled users. Drumm, et. al, 
IBM Technical Disclosure Bulletin, Vol. 27, No. 48, 
page 2528 (September 1984). However, the foregoing 
disclosure does not suggest a means of enabling a blind 
user to navigate about or locate objects on the computer 
display screen. 

Recently, in U.S. Pat. No. 5.223.828, issued Jun. 29. 
1993, entitled "Method and System for Enabling a Blind 
Computer User to Handle Message Boxes in a Graphi- 
cal User Interface", which is assigned to the assignee of 
the present application, a system has been proposed that 
permits a blind or visually impaired user to interact with 
message boxes within a graphical user interface. Each 
message box consists of an icon, explanatory text, and 
one or more "pushbuttons". The icon allows the user to 
identify visually the type of message. The text typically 
explains the situation and may provide assistance. The 
textual content may be a question or a statement. Push- 
buttons provided within the message box typically 
allow the user to interact with the message box. 

The system of U.S. Pat. No. 5,223,828 permits blind 
or visually impaired users to accommodate a message 
box by announcing the textual content of such a box 
when the message box first appears. Thereafter, the 
pushbuttons available to respond to the message box are 
announced in order from left to right. A homing signal 
was then provided for finding the message box. The 
homing signal increases in pitch as the mouse pointer 
approaches the message box. When the pointer enters 
the message box, the message box text and available 
pushbuttons are re-announced and the pointer is auto- 
matically moved to a default pushbutton. By using this 
system, a blind or visually impaired user may locate a 
message box within a computer system. 

Another system and method is disclosed in U.S. pa- 
tent application Scr. No. 08/022,788, filed Feb. 22, 1993, 
which is a continuation of abandoned U.S. patent appli- 
cation Scr, No. 07/746,840. filed Aug. 19, 1991, and 
entitled "Audio User Interface With Stereo and Fil- 
tered Sound Effects", which is assigned to the assignee 
of the present application. The system and method of 
application Ser. No. 07/746,840, permits a blind or visu- 
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ally impaired user to locate a mouse pointer or other 
graphical pointing device within the client area of a 
window within a graphical user interface by providing 
a stereo sound system and varying the intensity of the 
left and right audio channels to indicate the horizontal 
position of the mouse pointer. That system also pro- 
poses an increase in pitch of an associated sound to 
indicate the relative position of the pointer in the top/- 
bottom access of the client area of the window. 

Recently, in application Ser. No. 07/802,956. filed 
Dec. 5, 1991, entitled *'Method and System for Enabling 
Blind or Visually Impaired Computer Users to Graphi- 
cally Select Displayed Elements**, which is assigned to 
the assignee of the present application, there is disclosed 
a method and system that may be used to enable a blind 
or visually impaired computer user to graphically select 
a displayed graphic element within a computer system 
display, A unique identifiable audible signal is associ- 
ated with each displayed graphic clement. A moveable 
cursor clement or a pointer is displayed within the com- 
puter system display and a composite audible signal is 
periodically generated in response to the position of the 
moveable cursor clement. The composite audible signal 
preferably includes elements of each identifiable audible 
signal associated with each displayed graphic element 
within a pre-detcrmined radius of the location of the 
moveable cursor element. In one embodiment of that 
system and method, each displayed graphic element 
comprises multiple picture elements and the composite 
audible signal includes elements of each identifiable 
audible signal associated with each displayed graphic 
element having picture elements within a rotational 
sector of a circle having its origin at the moveable cur- 
sor clement and a radius equal to the pre-dctermincd 
radius. 

In certain graphical user interfaces, frequently used 
objects that have system wide application are posi- 
tioned in the background of the display screen, which is 
frequently called the "electronic desktop," Such objects 
may include printer icons and wastebin or shredder 
icons. The printer icons are provided so that a user may 
print a document or other objects by direct manipula- 
tion and shredder or wastebin icons are provided so that 
the user can delete a document or object by direct ma- 
nipulation. Those icons are positioned on the desktop so 
that they will be readily available to the user without 
the user having to open a window that may contain 
them. Users are also typically given the ability to cus- 
tomize their interfaces by placing various objects of 
their own choosing on their desktop. Certain operating 
systems adapted for graphical user interfaces, such as 
IBM OS/2 Presentation Manager, provide a facility 
that automatically arranges the icons on the desktop in 
rows. 

The prior art has provided certain tools by which a 
blind or visually impaired user may navigate within 
windows and find certain elements. However, there 
does not currently exist any tool by which a blind or 
visually impaired user can readily locate icons on the 
desktop. In the prior work on audio graphical user 
interfaces, the blind user has been able to wander about 
the screen in search of icons. When passing over an 
icon, certain sounds have been created or altered along 
with a verbal announcement of the identity of the icon 
using a text-to-spcech synthesizer. However, this ran- 
dom searching process can be tiresome and time con- 
suming to the user. 
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SUMMARY OF THE INVENTION 

The present invention provides a method and system 
that provides the user of the system with audio informa- 
5 lion regarding the position of the pointer on a display 
screen, wherein the screen has displayed thereon a 
background and at least one row of spaced apart icons. 
The system includes, in addition to a display screen, a 
pointing device for manually positioning a pointer on 
10 the screen, and left and right speakers. Generally, the 
system generates a first distinctive sound, which, in the 
preferred embodiment, is a tone, from at least one of the 
speakers whenever the pointer is located in the back- 
ground. Whenever the pointer is located in an icon row 
15 of the background that is occupied by an icon, the sys- 
tem generates a second distinctive sound, which, in the 
preferred embodiment, is a chord made up of the back- 
ground tone plus another tone that is distinctive of the 
particular icon row in which the pointer is located. 
20 Whenever the pointer is located in the background, but 
not in a row, the relative volume of the tone generated 
by the speakers is related to the relative right/left posi- 
tion of the pointer on the screen. Whenever the pointer 
is located in an occupied icon row, the distinctive chord 
25 is generated substantially only from the right speaker 
whenever the pointer is located to the right of the right- 
most icon in the row. Conversely, whenever the pointer 
is located in an occupied row and positioned to the left 
of the leftmost icon in the row, the sound is generated 
30 substantially only from the left speaker. Finally, when- 
ever the pointer is located in an occupied row, but 
between the leftmost and rightmost icons, the sounds 
generated from each speaker is substantially equal to 
that generated by the other. Also, whenever the pointer 
35 is located on an icon, the system generates a distinctive 
sound. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a pictorial view of a computer display 
40 screen with a plurality of rows of icons displayed 
thereon. 

FIG. 2 is a block diagram of a preferred system of the 
present invention. 
FIG. 3 is a block diagram of a preferred sound gener- 
45 ator of the system of the present invention. 

FIG. 4 is a flowchart showing a preferred software 
implementation of the present invention. 

DESCRIPTION OF THE PREFERRED 
5Q EMBODIMENT 

Referring now to the drawings, and first to FIG. 1, a 
computer display screen is designated generally by the 
numeral 11. Display screen 11 has displayed thereon a 
background 13 with a plurality of objects displayed 
55 thereon. The objects displayed on background 13 in- 
clude a window 15 and a plurality of labeled icons, 
including a "people" or address book icon 17 and a 
"shredder** icon 19. Display screen 11 also has dis- 
played thereon a pointer 21 that is moveable about the 
60 screen by means of a mouse (not shown in FIG. 1). A 
user of the system can operate the mouse to move 
pointer 21 to open, move, copy, or otherwise manipu- 
late objects displayed on screen 11. 
Background 13 is sometimes referred to as an "dec- 
65 tronic desktop." Certain objects, such as the labeled 
icons of FIG. 1, may reside on the user's desktop so that 
they are readily available to the user. Various operating 
systems, such as OS/2 Presentation Manager provide a 
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facility, which in the case of OS/2 Presentation Man- take the fonn of normal audio speakers, or headphones, 
ager is referred to as the "Desktop Organizer", that or the like. The system includes means, as will be de- 
arranges the icons on the desktop in rows. In FIG. 1, the scribed in detail hereinafter, for controlling the rcspjec- 
rows into which the icons of the desktop may be orga- tive volumes of the speakers to achieve stereo effects, 
nired are indicated by dashed lines, and they include, in 5 In the preferred embodiment of the invention, when- 
FIG. 1, a first row 23, a second row 25, a third row 27, ever pointer 21 is positioned in background 13 and not 
and a fourth row 29. The four rows of FIG. 1 are illus- in an occupied row. the respective volumes of the back- 
trativc only, and the system may be adapted to create ground tone produced by the speakers is related to the 
more or fewer rows than are illustrated in FIG. 1. It will left/right position of pointer 21 on screen 11, However, 
be recognized, of course, that the dashed lines of FIG. 10 whenever pointer 21 is in an occupied row. the stereo 
1 are for the purpose only of illustrating the present effect is altered such that whenever pointer 21 is to the 
invention, and do not actually appear on display screen left of the leftmost icon of the row, substantially all of 
11. In FIG. 1, rows 23 and 25 are occupied by icons, but the volume of the chord associated with the row is 
rows 27 and 29 are empty or unoccupied. Nevertheless, produced by the left speaker. Similarly, whenever 
rows 27 and 29 are available to receive icons of the 15 pointer 21 is located to the right of the rightmost icon of 
desktop and those rows exist in the system. the row, substantially all of the volume of the chord 
In the present invention, audio information is pro- indicative of the row is produced by the right speaker, 
vided to enable a blind or visually impaired user to Whenever pointer 21 is located in a row between the 
locate the various rows and the icons in the rows. It will leftmost and rightmost icons, but not on an icon, the 
be recognized that normally sighted users may also fmd 20 volume is balanced equally between the left and right 
the present invention advantageous in operating a sys- speakers. 

temwithagraphicaluscrinterface. Achart 30ofback- Referring now to FIG. 2, there is shown a block 
ground notes is located to the left of display screen 11. diagram of the system of the present invention. The 
The background notes chart 30 is for purposes of illus- system includes CPU hardware, which is indicated 
tration only, and does not actually appear when the 25 generally by, dashed block 33. Rutming on CPU hard- 
invention is in use. In FIG.- 1, whenever pointer 21 is ware is an operating system 35, which includes presen- 
located in background 13, as opposed to, for example, in tation logic 37. Presentation logic 37 manages the pre- 
window 15, or on one of the labeled icons, a back* sentation of text and graphic information on the com- 
ground sound is produced. In the preferred embodi- puter display screen. A plurality of application pro- 
mem, the background sound is a single tone, which for 30 grams 39 are shown nmning on operating system 35. 
purposes of illustration, is an F2 according to the Ameri- The system includes video interface logic and hardware 
can Standard pitch, adopted by the American Standards 41, which supplies video information to a video monitor 
Association in 1936, which has a frequency of 87.31 43. 

Hertz. Whenever pointer 21 is located in background The system includes a keyboard 45 and a mouse 47, 
13, the system produces the note F2. If pointer 21 is 35 which allow the user to input data and operate the 
located somewhere other than background 13, the sys- system. The system also includes query code, which is 
lem produces an appropriate sound effect. For example, designated generally by the numeral 49. As will be 
if pointer 21 is located in window 15, the system pro- described in greater detail, query code 49 queries the 
duces the sound effects shown and described in U.S. system as to the current position of pointer 21. Query 
patent application Ser. No. 07/746.840, filed Aug. 19. 40 code 49 also provides text information to text-to-speech 
1991. Similarly, if pointer 21 were positioned on shred- hardware 51 via RS232 port 53 and sound information 
der icon 19, the system would announce, by text-to- to sound generation software and hardware, shown 
speech or recorded speech, "shredder". generally at block 55. The speech and sound informa- 
Referring still to FIG. 1, whenever pointer 21 is posi- tion is provided to the user by speakers or headphones 
tioned in an occupied row, the system produces, in 45 S7 8Text-to-speechpi ardware 51 is commercially avail- 
addition to the background note F2, an additional note abl Jr^ ^ 

that makes a distinctive chord that identifies each row. Referring now to FIG. 3, there is shown a block 

Whenever a row is not occupied, the system produces diagram of the sound generation software and hardware 

only the background note F2 and not the additional 55 of the present invention. Sound generation hardware 

note. Thus, since first row 23 is occupied, the system 50 55 includes at least two oscillators, which are desig- 

produces simultaneously the notes F2 and D3, which nated by the numeral 59. Oscillators 59 include at least 

makes a chord. Similarly, since row 25 is occupied, the a first oscillator, which may be fixed or variable, that 

system produces the chord comprising the notes F2 and produces the background note, which in the preferred 

C3. Since third row 27 and fourth row 29 are unoccu- embodiment is F2. Oscillators 59 also includes a variable 
pied, the system produces only the note F2. However, if 55 second oscillator that may be controlled to produce the 

rows 27 or 29 were occupied, then the appropriate secondary notes that combine with background note F2 

chord would be produced as set fonh in table 30 of to produce the distinctive chords for occupied rows, 

background notes. The tones produced by oscillators 59 are summed by a 

In the preferred embodiment of the invention, the summing circuit 61. 
system produces, in addition to tonal information with 60 The sound generation software includes outputs that 

respect to the location of pointer 21, which is related are enclosed in dashed rectangle 55a, Sound generation 

generally to the vertical, or top/bottom position of software outputs include an oscillator control 63, which 

pointer 21 in background 13, stereo information respect- turns on and off the various oscillators 59 and controls 

ing the horizontal or left/right position of pointer 21 in the frequency of the various variable frequency oscilla- 
background 13. 65 tors of oscillators 59. 

Accordingly, in the system of the preferred embodi- The output from summing circuit 61 is split at 67 into 

ment of the invention, a pair of laterally spaced apart left and right channels. A left amplitude control 69 

stereo speakers are provided. The stereo speakers may controls a variable attenuator 71 in the left channel and 
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a right amplitude control 73 controls a variable attenua- the amplitude of the right channel of the sound genera- 
tor 75 in the right channel. The output from variable tor to zero and the amplitude to the left channel to one 
attenuator 71 is amplified by an output amplifier 77 and and returns to block 85 to continue monitoring pointer 
the audio signal is produced at left speaker 79. Similarly, position. If, on the other hand, the pointer is not left at 
the output from variable attenuator 75 is amplitude by 5 the leftmost icon, the system tests at decision block 113 
an output ^plificr 81 and produced as an audio signal whether or not the pointer is to the right of the right- 
at speaker 83. j • w n u ^ ^^^^ the system sets the amplitude to the 
Rcfemng now to FIG. 4, there is shown a flowchart • « . ^/„„ j „ » , . 
- r*. ... lr*i. jf^t. right channel ol the sound generator to one and the 
of a preferred embodiment of the query code of the . "^t'., i-f» i * .^^^^n^m ^ 
present invention. First, the pointer position (Xptr, jo ^^P^^^^f^ *^ 

Vptr) is queried at block 85. Tlien, the system tests, at retun« to block 85 to continue monn^^^^ 

decision block 87, whether or not the pointer is posi- decision blocks 109 and 113 the pomier is 

tioned on the background. If, at decision block 87, the ^^^^ leftmost icon nor to the right of 

pointer is on the background, then, at block 89, the nghtmost icon, the system sets the output to both 
frequency of oscillator one is set to the background 15 the nght and left channels of the sound generator to 0.5 

tone. If, on the other hand, the pointer is not on the at ^^^ck 117 and returns to block 85 to continue moni- 

background, the system tests at decision block 91 toring pointer position. 

whether or not the pointer is on an icon; if it is, then, as In operation, a user can quickly run the pointer up or 

generally indicated at block 93, the system produces the down the background and identify the number and 

appropriate icon sound, which in the preferred erabodi- location of the rows of icons on the desktop. The chords 

ment is an announcement of the name of the icon by produced when the pointer is on a row are distinctive 

text-to-speech. After the system has made the icon and easily recognized by the user. After the user has 

sound at block 93, the system returns to block 85 to located the rows, the user can quickly scan each row to 

continue monitoring the position of the pointer. If, at find the appropriate icon. If the user hears the chord 
decision block 91, the pointer is not on an icon, then the ^5 from the right speaker, the user knows that all of the 

system performs other processing, indicated generally i^ons in the row are toward the left. The user can move 

at block 95. For example, if the pomter is on a window. ^^e pointer toward the left and hear the announcement 

the system PJ<^^^^^' P°^l'^°" ''^?o'^oo^^ ^^^^ '^^^ ^hen the pointer is between 

application Scr. No. 07/746.840. Hied Aug^ 19. 1991. j^^^^^^^ ^^^^^^^ . >^ 

Refernng again to ^^^^.f ^^^^^^^^ 30 between the two speakers. Tlius. the user knows that 

on the background and after oscillator one has been set . w ^if. .1. » ^ \j . . ^ ^ 

to the background tone at block 89, the system tests at .'l^^' u 

block 97 whether or not the pointer is on a row. If the ^^'^'l^^ ''''' "J^^^^^ ^0 ^yo"d 

pointer is on a row. then the system tests at decision the leftmost icon of the row, the balance of the chord 

block 99 whether or not the row is occupied. If, at shifts to the left speaker, thereby mforming the user that 

decision block 97, the pointer is not on a row, or if, at ^5 there are no more icons m the row. If the user moves 

decision block 99, the pointer is on an unoccupied row, the pointer back toward the right, the balance will shift 

the system calculates, at block 101, the pointer position to the middle and then to the right. Of course, if there is 

relative to the display extent along the X axis by the only one icon in a row, the balance will be cither all 

formula: from the right or all from the left depending on whether 

40 the pointer is to the right or left of the single icon. 

Xptr From the foregoing, it may be seen that the system 

^ ~ Xr - Xt and method of the present invention provides a blind or 

visually impaired user with audio information sufficient 

Where: Xr is the X coordinate of the right edge of the to enable the user to locate quickly and easily icons 

background; and XI is the X coordinate of the left displayed in rows on the background of a graphical user 

edge of the background. interface. The present invention may also find use 

The denominator of the foregoing formula is basically among normally sighted users who desire additional 

the width of the display screen. Thus, Px is a number sensory input. While the invention has been particularly 

from zero to one that reflects the position of the pointer j^own and described with reference to a preferred em- 

along the X or left/right axis of the display screen. 50 j^i^^^^^ ^^ose skilled in the art will understand that 

After the system has calculated the relative pointer ^^^^ous changes may be made in form and detail with- 

position at block 101 the system outputs the relative ^ ^^^^ ^^^^ ^^^^^ .^^^^^.^^ 

amplitudes of the nght and left channels to the sound ^j^^j claimed is- 

generator at block 103 and returns to continue monitor- - . J,^, m: r 

ing pointer position at block 85. 55 \ ^ of providmg the user of a computer 

If^t decision blocks 97 and 99, the pointer is on an system including a display screen, a pointmg device for 

occupied row. then the system, at block 105, derives the "^^"IJ^^y positioning a pointer on said screen, a left 

row number corresponding to the pointer's position, '^P^^^^' « ^P^^^^^y- ^»^h audio mformation 

Then, at block 107, the system sets oscillator number regarding the position of said pointer on said screen, 

two to the tone corresponding to the row number. Re- 60 wherein said screen has displayed thereon a background 

ferring briefly to FIG. 1. if pointer 21 were in first row ^t least one row of spaced apan icons, said method 

23, oscillator two would be set to the tone Dj. The comprising the steps of: 

tones related to the various row numbers can be main- generating a first sound from at least one of said 

tained in a look-up table. ' speakers whenever said pointer is located on said 

After the system has set oscillator number two to the 65 background; and 

tone corresponding to the row number the system tests generating a second sound from at least one of said 

at decision block 109 whether the pointer is to the left of sp)eakers whenever said pointer is located on said 

the leftmost icon. If it is, the system, at block 111 sets row. 
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2. The method as claimed in claim 1> therein said thereon a background and at least one row of spaced 
first sound is a first tone. ap^rt icons, said apparatus comprising: 

3. The method as claimed in claim 2, wherein said » speaker and a right speaker; 

second sound is a chord. ^^^^ generating a first sound from at least one of 

4. The method as claimed in claim 3, wherein said ' bff ^ou^d^^ d"^^ ^" 
chord includes said first tone and a second tone. r^, „r\. . * , . . , 

- ^. J 1 • J • 11 u • means for generatmg a second sound from at least one 

5. The method as claimed m claim 1, whercm the cqM c«oai.«^e«3, j • * • i ^ 

. 'j. J ot said speakers whenever said pomter IS located on 

relative volume of said first sound generated by said ^^^j 

speakers is related to the relative horizontal position of lo 12. The apparatus as claimed in claim 11, whereiri 

said pointer on said screen. said means for generating said first sound includes 

6. The method as claimed in claim 1, wherein said means for generating a first tone. 

row of icons is oriented horizontally on said screen and 13. The apparatus as claimed in claim 12, wherein 

said second sound is generated substantially only from said means for generating said second sound includes 

said right speaker whenever said pointer is located on 15 means for generating a chord, 

said row and to the right of the rightmost icon in said The apparatus as claimed in claim 13, wherein 

said chord includes said first tone and a second tone, 

7. The method as claimed in claim 6, wherein said '^}^^ apparatus as clainjed in claim 11, including 
second sound is generated substantially only from said "^^^ controUmg the relative volume of Mid first 
, - , . . . ■ * • I r J J 20 sound generated by said speakers relative to the hori- 
left speaker whenever said pomter is located on said ^^^^^^ po^on said screen. 

row and to the left of the leftmost icon in said row. ^pp^^^^^ ^^^^ ^^^^^^ 

8. The method as claimed m claim 7, wherein said ^ow of icons is oriented horizontally on said screen 
second sound is generated from each of said speakers at and said second sound is generated substantially only 
substantially equal volume whenever said pointer is 25 from said right speaker whenever said pointer is located 
located on said row and between the leftmost and right- on said row and to the right of the rightmost icon in said 
most icons, row. 

9. The method as claimed in claim 1, including the 17. The apparatus as claimed in claim 16, wherein 
step of generating a distinctive sound whenever said said second sound is generated substantially only from 
pointer is located on an icon, 30 said left speaker whenever said pointer is located on 

10. The method as claimed in claim 1, wherein more ^^^^ leftmost icon in said row. 
than one row of icons is displayed on said screen and }f The apparatus as claimed m claim 17, wherein 

. , J- - r *• j-rr . said sccond sound IS generated from each of said socak- 

mc udine the step of generating a different sound for * 1. * n 11 ^ w« v* ^ys^ai^ 

^ r crs at substantially equal volume whenever said pointer 

each or said rows. ^ , 35 is located on said row and between the leftmost and 

11. Apparatus for providing the user of a computer rightmost icons. 

system, including a display screen and a pointing device 19. xhe apparatus as claimed in claim 11, including, 

for manually positioning a pointer on said screen, with means for generating a distinctive sound whenever said 

audio information regarding the position of said pointer point is located on an icon, 

on said screen, wherein said screen has displayed 40 * * ♦ • » 
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